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Preface 


The  purpose  of  this  study  was  to  develop  a 
multicriteria  decision  model  to  aid  the  Air  Force  in 
prioritizing  Installation  Restoration  Program  projects.  The 
need  for  research  on  this  subject  has  been  expressed  by  many 
individuals,  ranging  from  project  programmers  at  the  base 
level,  to  environmental  restoration  experts  at  the  Major 
Command  and  Air  Staff  levels,  to  the  Deputy  Assistant 
Secretary  of  the  Air  Force  for  the  Environment . 

This  research  was  accomplished  using  a  Delphi  process 
and  a  multiattribute  utility  theory  questionnaire.  We  would 
like  to  gratefully  acknowledge  the  time  and  effort  expended 
by  those  who  participated  in  our  questionnaires.  We  would 
also  like  to  thank  our  readers,  Lt .  Colonels  Mark  Goltz  and 
Mike  Shelley,  for  their  insightful  guidance  on  the 
environmental  aspects  of  our  work,  and  Dr.  Guy  Shane  for  his 
enthusiastic  support,  his  assistance  with  the  Delphi 
technique,  and  his  editorial  efforts.  A  special  thanks  must 
also  go  to  our  faculty  advisor.  Dr.  Yupo  Chan,  who  patiently 
endured  the  process  of  teaching  us  multicriteria  decision 
making.  Without  his  dedication,  support,  and  guidance,  this 
research  effort  would  not  have  been  possible. 

An  Air  Force  Institute  of  Technology  thesis  is  a  very 
time  consuming  and  mentally  taxing  effort.  Many  hours  are 
spent  reading  background  material,  gathering  and  analyzing 
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data,  and  documenting  results.  Often  our  scholastic 
responsibilities  required  that  we  spend  more  time  with  our 
books  than  with  our  loved  ones.  For  their  support  and 
understanding,  we  would  like  to  thank  our  families. 

Gene  would  especially  like  to  thank  his  wife  Tami  for 
maintaining  the  family  single-handedly  during  the  times  he 
was  preoccupied  with  schoolwork.  He  also  promises  to  spend 
more  time  with  his  children;  Katie,  Erin,  Craig,  Kelsey,  and 
Jenni;  now  that  we  have  completed  our  work. 

Tom  would  like  to  give  special  thanks  to  his  parents, 
Don  and  Mary  Ellingson  for  their  confidence,  love,  and 
teaching,  not  just  during  the  past  year  and  a  half,  but  over 
the  last  twenty-seven  years.  Also,  a  very  special  thanks  to 
his  fiance  Donna  Esterbrook  for  her  contributions  to  this 
effort,  for  enduring  the  time  spent  apart,  and  most  of  all, 
for  promising  to  share  her  life. 


T.  Gene  Gallogly 
Thomas  E.  Ellingson 
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Cleaning  up  hazardous  and  toxic  wastes  left  from  past 


activities  has  become  one  of  the  major  goals  of  the 
Department  of  Defense.  Within  the  Air  Force,  individual 
major  commands  (MAJCOMs)  annually  review  numerous  projects 
submitted  under  the  Defense  Environmental  Restoration 
Program  (DERP)  and  forward  these  projects  to  HQ  USAF/CEVR 
for  funding.  Until  recently,  adequate  resources  have  been 
available  to  fully  fund  all  projects.  However,  within  the 
last  two  years,  funding  shortfalls  have  been  experienced, 
and  projects  have  received  only  partial  funding.  The  Air 
Force  needs  a  decision  support  system  that  will  assist  in 
determining  funding  priorities  for  these  projects. 

This  thesis  presents  two  possible  methodologies  and 
resulting  multicriteria  decision  models  for  determining 
funding  priorities  within  the  constraints  of  a  limited 
budget.  The  first  method  utilizes  a  Delphi  procedure  to 
elicit  pertinent  criteria,  weights,  and  utilities  from  DERP 
experts.  The  second  method  utilizes  multiattribute  utility 
theory  to  determine  the  preference  structure  of  a  single 
decision  maker.  Each  method  results  in  a  working  model 
which  can  be  used  to  prioritize  DERP  projects.  Both  models 
resulted  in  risk  being  the  primary  decision  criterion. 
Additionally,  the  Delphi  model  also  utilized  cost,  community 
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DEVELOPMENT  OF  A  DECISION  SUPPORT  SYSTEM 
FOR  PRIORITIZING  AIR  FORCE 
ENVIRONMENTAL  RESTORATION  PROGRAM  PROJECTS 

I.  Introduction 

The  toxic  wastes  of  New  York's  Love  Canal,  Kentucky's 
Valley  of  the  Drums,  and  Missouri's  Times  Beach  brought  to 
light  serious  threats  to  the  public's  health  from  past 
hazardous  waste  disposal  practices  and  spills  (EPA, 

1990c:l).  In  1975,  five  years  before  the  passage  of  the 
Comprehensive  Environmental  Response,  Compensation,  and 
Liability  Act  (CERCLA) ,  the  Department  of  Defense  (DoD)  took 
note  of  the  public  furor  caused  by  these  events  and  began 
the  Defense  Environmental  Restoration  Program  (DERP)  (Air 
Force,  1992:23).  Given  the  heightened  interest  in  the 
environment  and  the  number  of  base  closures  as  a  result  of 
defense  drawdowns,  DERP  has  become  one  of  the  Armed 
Service's  top  priorities  (Baca,  1992:18). 

From  its  inception,  DERP  has  evolved  to  keep  pace  with 
legislative  developments,  public  pressures,  and  new 
technologies.  CERCLA  and  its  amendments,  similar  state 
laws,  and  several  DoD  and  Environmental  Protection  Agency 
(EPA)  regulations  set  forth  the  overall  objectives, 
procedures,  and  cleanup  standards  for  the  program.  While 
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these  laws  and  regulations  establish  the  actions  that  will 
be  required,  funding  determines  the  rate  at  which  the 
cleanups  progress. 

DERP  funds  are  distributed  from  a  special  Congressional 
appropriation  whose  growth  has  paralleled  the  popular 
environmental  movement.  Until  recently,  these  funds  have 
been  adequate;  however,  new  sites  are  continually  uncovered, 
the  number  of  studies  and  remediation  projects  under  DERP 
continue  to  grow,  and  the  program  has  evolved  from  the 
investigative  stages  to  the  more  costly  remediation  stages. 

With  the  heightened  awareness  and  effort  has  come  an 
increased  competition  for  available  funds.  Last  year,  the 
Air  Force  was  able  to  fund  only  45  percent  of  its  program 
(Owendoff ,  1993a) .  While  various  means  to  allocate  these 
funds  have  been  proposed,  none  of  the  schemes  adequately 
account  for  the  multiple  program  objectives  and  the 
regulatory  commitments  made  at  the  various  administrative 
levels.  Instead,  funds  are  allocated  to  the  Major  Commands 
(MAJCOMs )  based  on  each  MAJCOM's  percentage  of  total  costs 
for  all  projects  meeting  basic  screening  parameters 
{Owendoff,  1993a) .  This  decision  rule  forces  DERP  managers 
to  delay  critical  work  at  some  bases  while  less  important 
work  proceeds  at  another  base  under  a  different  MAJCOM.  The 
situation  is  exacerbated  as  competition  increases. 
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Given  the  funding  dilemma,  Mr.  Gary  Vest,  the  Deputy 


Assistant  Secretary  of  the  Air  Force  for  Environment, 

Safety,  and  Occupational  Health,  requested  development  of  a 
new  decision  support  system  (DSS)  (Vest,  1992) .  A 
multicriteria  decision  model  (MCDM)  in  support  of  the  DSS 
will  help  establish  a  distinct  funding  hierarchy  among  DERP 
project  alternatives  at  every  stage  in  the  cleanup  process. 
The  resulting  priorities  must  be  consistent  with  CERCLA, 
state  laws,  and  the  governing  regulatory  objectives. 
Furthermore,  the  funding  priorities  should  be  acceptable  to 
the  MAJCOMs  who  have  been  delegated  responsibility  for  the 
program.  Finally,  the  model  must  satisfy  Congress,  the  EPA 
and  state  regulatory  agencies,  and  the  affected  public  who 
decide  how  large  the  total  DERP  budget  will  be,  who  help 
decide  what  the  installation  priorities  are,  and  who  judge 
the  results  of  Air  Force  efforts. 

R£se.ai:cti  Objectives 

The  purpose  of  this  research  is  to  support  development 
of  a  new  DSS  for  determining  DERP  priorities.  In  order  to 
do  this,  the  research  will  be  designed  to  accomplish  two 
objectives.  First,  the  goals  of  the  DERP,  their  relative 
importance,  and  their  relationships  with  one  another  will  be 
determined.  Second,  these  findings  will  be  used  to  develop 
an  MCDM  to  support  Air  Force  decision  makers. 


The  first  objective  will  be  accomplished  by  means  of  a 
literature  review  and  surveys  of  key  DERP  and  regulatory 
personnel.  The  following  questions  should  be  answered  to 
fully  support  this  objective: 

1)  What  are  the  objectives  of  CERCLA,  similar  state 
laws,  and  DoD  and  EPA  regulations  with  regard  to  the  DERP? 

2)  Which  DERP  project  criteria  are  most  important  to 
meeting  these  objectives  and  how  are  they  measured? 

3)  How  should  inherent  project  uncertainties  be 
addressed? 

4)  How  can  a  comparison  be  made  among  multi-phase, 
multi-year  projects  with  different  cost  and  benefit  streams? 

The  answers  generated  from  the  literature  review  and 
survey  will  be  incorporated  into  a  suitable  MCDM  to  satisfy 
the  second  objective.  As  a  result,  the  MCDM  should  assist 
in  making  decisions  which  are  less  arbitrary,  make  use  of 
the  full  power  of  discriminating  criteria,  and  most 
importantly,  allocate  funding  to  deserving  projects  more 
equitably. 

SCQPe/kiwitfltiQR 

Since  future  requirements  are  expected  to  grow  much 
faster  than  DERP  appropriations,  a  new  means  of  prioritizing 
projects  is  urgently  needed.  As  such,  the  research  will  not 
devote  time  to  developing  new  means  to  measure  progress  with 
DERP  objectives.  Rather,  an  attempt  will  be  made  to  reach  a 
consensus  among  experts  on  the  most  suitable  existing 
measures  to  use  in  the  model  (a  partial  response  to 


4 


question  2  above) .  Additionally,  the  MCDM  will  not  be 
developed  to  help  choose  among  alternative  methods  for  site 
remediation.  At  this  stage  in  the  research,  existing 
systems  and  expert  judgment  must  be  relied  on  to  make  site- 
specific  remediation  technology  decisions  and  to  weigh  the 
uncertainties  involved  in  the  choices.  Finally,  validating 
the  MCDM  with  an  existing  ranking  of  actual  projects  will 
not  be  attempted.  Key  parameter  measures  may  not  be 
available  in  existing  programming  documents,  and  the  time  to 
research  or  develop  them  will  not  be  available. 

While  these  limiting  features  will  likely  raise 
suspicions  as  to  the  validity  of  the  resulting  MCDM,  the 
overall  research  objective  will  be  met.  The  MCDM  will 
provide  an  immediate,  equitable  means  to  allocate  funds  and 
will  give  DoD  a  starting  point  upon  which  to  build  support 
for  their  prioritization  system,  solicit  improvements,  and 
respond  fully  to  the  four  questions  posed  previously. 

Organisation  of  the  _Research,_Rgp<?rt 

The  first  step  in  developing  an  appropriate  MCDM  for 
prioritizing  DERP  projects  is  to  establish  an  understanding 
of  the  program,  its  written  objectives,  and  previous 
research  into  similar  problems.  The  literature  review 
accomplishes  this  purpose.  Following  the  literature  review, 
a  description  of  the  methodologies  used  to  gather  relevant 
criteria  for  the  prioritization  problem  and  the  preference 
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structures  used  to  analyze  and  compare  the  criteria  are 
discussed.  Additionally,  the  procedures  employed  to  develop 
appropriate  MCDMs  for  the  decision  support  systems  are 
addressed.  The  research  findings  and  analysis  are  reported 
in  Chapter  Four,  followed  by  the  conclusions  and 
recommendations  for  further  study  in  Chapter  Five. 
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II.  Literature  Review 

Cleaning  up  hazardous  and  toxic  wastes  left  from  past 
Department  of  Defense  (DoD)  activities  is  receiving  national 
attention.  Funding  for  cleanups  has  grown  from  just  $150 
million  in  fiscal  year  1984  to  more  than  a  billion  dollars 
in  recent  years  {DoD,  1992:1;  West,  1991:8).  More  and  more 
contaminated  sites  are  being  identified,  and  the  number  of 
studies  and  remediation  projects  in  the  Defense 
Environmental  Restoration  Program  (DERP)  continues  to  grow. 
With  the  heightened  awareness  and  efforts  has  come  an 
increased  competition  for  available  funds.  This  research  is 
aimed  at  developing  a  multicriteria  decision  model  (MCDM) 
for  use  in  a  new  decision  support  system  (DSS)  for 
determining  DERP  funding  priorities  among  the  competing 
projects . 

Two  objectives  must  be  met  in  order  to  develop  the  DSS. 
First,  the  goals  of  DERP,  their  relative  importance,  and  the 
relationships  between  them  must  be  determined.  Second, 
these  findings  must  be  used  to  develop  an  MCDM  to  support 
Air  Force  decision  makers. 

A  thorough  understanding  of  the  program  is  required 
before  the  first  objective  can  be  met;  therefore,  the 
program's  background  was  researched.  The  applicable 
legislation,  a  brief  history  of  the  events  that  led  up  to 
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their  passage,  and  the  EPA's  interpretations  of  the 
legislative  mandates  were  each  researched  to  determine  the 
goals  and  objectives  of  the  program.  Next,  DoD  and  Air 
Force  programs  were  studied  to  provide  insight  into  the 
scope  of  the  problem.  They  were  also  analyzed  to  determine 
if  the  stated  objectives  for  their  programs  were  consistent 
with  the  legislation,  and  if  the  current  project  approval 
and  prioritization  system  would  fulfill  those  objectives. 
After  establishing  program  objectives,  common  restoration 
methods  were  discussed  to  provide  a  better  understanding  of 
the  current  state  of  technology,  aid  in  developing  criteria 
and  measures  to  distinguish  among  project  alternatives,  and 
understand  project  uncertainties  better. 

To  satisfy  objective  two,  the  answers  generated  from 
this  literature  review  and  surveys  of  experts  will  be 
incorporated  into  a  suitable  MCDM.  A  brief  review  of 
decision  models  and  decision  making  theory  was  conducted  to 
gain  an  appreciation  of  this  task.  A  more  thorough  review 
of  MCDMs  will  be  presented  in  the  methodology  section. 
Likely  parameters  for  the  model  were  also  reviewed.  Since 
it  appeared  that  reducing  risk  was  of  paramount  importance, 
risk  as  a  decision  parameter  was  researched  in  detail. 

Other  seemingly  important  factors  such  as  cost- 
effectiveness,  feasibility,  and  the  value  of  innovative 
technologies  were  also  analyzed  to  aid  in  determining  their 
relative  importance  to  the  proposed  model. 
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Fina’ly,  to  complete  the  literature  review,  similar 
research  done  by  others  was  examined.  Several  studies 
related  to  prioritizing  work  based  on  involving  the  public, 
reducing  risk,  funding  the  worst  sites  first,  and  focusing 
on  incremental  improvements  were  found  to  be  relevant  to 
this  MCDM. 

Background 

DERP  has  not  existed  very  long,  but  it  has  received 
much  attention  in  this  decade  of  environmental  awareness. 
Both  Congress  ar.d  the  public  expect  progress  for  the  money 
they  are  investing  in  the  program,  and  the  Air  Force's 
prioritization  methods  and  subsequent  allocation  of  funds 
need  to  address  their  concerns.  Therefore,  a  review  of  the 
program’s  inception  and  an  understanding  of  where  it  stands 
today  are  discussed  below: 

Legislation.  The  first  significant  attempt  at 
regulating  threats  to  the  public's  health  and  the 
environment  from  past  hazardous  waste  disposal  sites  was  the 
Comprehensive  Environmental  Response,  Compensation,  and 
Liability  Act  (CERCLA)  of  1980  [Public  Law  96-510,  42 
United  States  Code  (USC)  9601-9657] .  This  law,  commonly 
referred  to  as  Superfund,  authorized  the  government  to 
respond  to  releases  or  threatened  releases  of  hazardous 
substances  that  could  endanger  the  public  or  the 
environment.  It  also  established  a  means  to  force 
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responsible  parties  to  pay  for  cleanups  and  provided  a  $1.6 
billion  fund  to  clean  up  the  Nation's  highest  priority 
sites.  (EPA,  1990c:l) 

According  to  Don  Clay,  Assistant  Administrator  for  the 
Solid  Waste  and  Emergency  Response  (OSWER)  office  of  the 
EPA,  Superfund  had  six  goals: 

1)  Make  the  sites  safer.  Control  immediate  threats 
and  cleanup  the  worst  sites  first. 

2)  Make  sites  cleaner.  Quickly  implement  and  improve 
long-term  cleanups. 

3)  Strengthen  enforcement  and  maximize  actions  by 
responsible  parties. 

4)  Encourage  innovative  restoration  technologies. 

5)  Involve  the  community. 

6)  Communicate  results  to  the  public.  (EPA,  1990c :V) 

The  Superfund  Amendments  and  Reauthorization  Act  (SARA) 

of  1986  was  passed  when  it  became  apparent  that  CERCLA  was 
not  going  to  accomplish  its  objectives.  SARA  broadened 
CERCLA,  extended  the  authorization  for  five  years,  and  added 
$8.5  billion  to  the  cleanup  fund.  It  also  emphasized  the 
use  of  alternative  or  resource  recovery  techniques  and  the 
need  for  permanent  solutions.  (EPA,  1990c :2) 

Sovereign  immunity  is  often  waived  in  federal  statutes 
to  require  federal  agencies  to  comply  with  the  procedural 
and  substantive  requirements  of  state  and  local  regulations 
(Spectrum,  1990:6).  CERCLA  Section  107(g)  waives  sovereign 
immunity  for  federal  facilities.  This  section  was  later 
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reinforced  by  SARA  Section  120  (EPA,  1988:11-6).  Executive 
Order  12088  also  dictates  that  agencies  comply  with 
applicable  pollution  control  standards  and  that  they 
cooperate  with  state  and  local  agencies  to  control  and  abate 
pollution  (EPA,  1988:11-1).  DoD  has  taken  these  mandates 
seriously,  and  former  Defense  Secretary  Cheney  committed  DoD 
to  an  environmental  leadership  role  (Spectrum,  1990:1). 

Air  Force  and  DoD  cleanup  activities  fall  under  the 
auspices  of  DERP,  which  was  established  by  Congress  as 
Section  211  of  SARA  (EPA,  1988:11-7)  under  Title  10,  USC 
2701-2707  and  2810  (Air  Force,  1992:1).  DERP  consists  of 
three  basic  elements.  The  firsc  element,  the  Installation 
Restoration  Program  (IRP)  was  established  to  identify, 
investigate,  and  clean  up  contamination  at  federal 
installations.  Its  primary  purpose  is  to  reduce  public 
health  risks  by  cleaning  up  hazardous  wastes  associated  with 
past  practices  and  to  restore  affected  natural  resources  for 
future  use.  Approximately  94%  of  the  $1,165  billion  in 
fiscal  year  1991  was  used  for  the  IRP  (DoD,  1992 :i)  The 
second  element  consists  of  other  hazardous  waste  operations. 
This  program  funds  hazardous  waste  reduction  initiatives  to 
include  equipment,  costs  for  procedural  changes,  and 
research  and  development  costs.  It  also  may  fund  unexploded 
ordnance  detection  and  range  clearance  operations.  The 
third  and  final  element  is  the  building  demolition  and 
debris  removal  program.  Under  this  program,  unsafe 
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buildings  and  structures  can  be  torn  down  and  removed.  (Air 
Force,  1992:2) 

CERCLA,  as  amended  by  SARA,  and  the  Clean  Water  Act 
(CWA)  both  gave  rise  to  the  National  Contingency  Plan  (NCP) . 
The  NCP  was  rewritten  by  the  EPA  under  the  authority  of 
Executive  Order  12580.  The  NCP  provides  implementing 
guidance  and  procedures  the  Air  Force  must  follow  to  comply 
with  CERCLA,  SARA,  and  the  CWA  (EPA,  1990a : 8813-14) .  Of 
particular  interest,  the  NCP  sets  forth  nine  criteria  for 
the  selection  of  remedial  alternatives: 

1)  Will  the  proposed  alternatives  adequately  protect 
human  health  and  the  environment? 

2)  How  well  do  che  alternatives  meet  federal  and  state 
Applicable  or  Relevant  and  Appropriate  Requirements  (ARARs)? 

3)  What  is  the  residual  risk  at  the  site  after 
completion  of  the  proposed  remedial  actions?  This  criterion 
assesses  long-term  effectiveness  of  the  remedy. 

4)  Do  the  proposed  alternatives  permanently  and 
significantly  reduce  the  toxicity,  mobility,  or  volume  of 
the  hazardous  wastes? 

5)  How  well  do  the  actions  address  human  health  and 
environmental  effects  during  implementation  and  until  the 
cleanup  objectives  are  met?  This  criterion  addresses  short¬ 
term  effectiveness  which  compliments  the  long-term 
effectiveness  assessed  in  the  third  criterion. 

6)  Can  the  alternatives  be  readily  implemented?  Are 
they  technically  feasible,  administratively  feasible,  and 
will  the  necessary  personnel  and  materials  be  available  when 
required? 

7)  Are  the  alternatives  cost-effective  in  relation  to 
the  other  criteria? 

8)  Are  the  alternatives  acceptable  to  the  affected 
state? 
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9)  Are  the  alternatives  acceptable  to  the  affected 
community,  including  the  public,  other  groups,  and  other 
potentially  responsible  parties  (PRPs)? 

The  NCP  states  that  the  lead  agency  (the  Air  Force  on  its 
installations)  must  balance  the  tradeoffs  among  the 
alternatives  with  respect  to  these  nine  criteria.  The 
alternative  that  protects  human  health  and  the  environment, 
complies  with  the  ARARs,  and  best  meets  the  other  criteria 
is  selected  as  the  preferred  alternative.  (EPA,  1990a:8719- 
8724) 

Aside  from  explicit  legislation  and  regulations,  EPA 
policy  directives  also  provide  insight  on  where  cleanup 
priorities  should  be  set.  In  a  recent  directive,  OSWER 
noted  that  the  top  priority  for  Superfund  is  to  reduce 
imminent  risks,  and  that  the  site  closure  and  deletion  from 
the  National  Priorities  List  (NPL)  is  secondary  to  risk. 
Efficiency,  effectiveness,  and  equitability  were  espoused  by 
OSWER  as  its  key  principles.  In  the  same  directive,  OSWER 
noted  that  federal  facility  enforcement  was  extremely 
important  because  of  their  program's  high  visibility,  the 
interest  shown  by  the  public,  the  significant  threats  posed, 
the  impact  of  base  closings,  and  the  magnitude  of  the 
resources  needed  to  oversee  their  operations.  Their  goals 
with  regard  to  federal  sites  are  to  improve  efficiency 
through  accelerated  response,  integrate  the  Resource 
Conservation  and  Recovery  Act  (RCRA)  and  CERCLA  through 
federal  facility  agreements  (FFAs),  use  innovative 
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technologies  and  pollution  prevention  principles,  comply 
with  mandated  time  frames,  and  enhance  oversight  of  base 
closure  actions.  (Office  of  Emergency  and  Remedial 
Response,  1992:1-1-2,  17-18) 

In  addition  to  the  watchful  eyes  of  the  EPA  and  state 
agencies,  SARA  Section  120  includes  a  provision  enabling 
private  citizens  to  sue  the  government  (EPA,  1988:11-6). 

SARA  Section  310  states  that  these  suits  may  be  brought 
against  a  party  or  the  responsible  officials  who  violate  a 
requirement  or  an  order  under  Super fund.  Furthermore,  RCRA 
Section  7002  authorizes  citizen  suits  to  force  appropriate 
actions  addressing  hazardous  wastes  when  they  pose  a  danger 
to  human  health  or  the  environment  (Lucero,  1989:81). 

hix  Farqg_Ina£al!atiQh  Res  taxation  Program-  According 
to  Mr.  Gary  Vest,  the  Air  Force  has  approximately  4,000 
contaminated  sites  in  their  program.  To  meet  the  Air  Force 
Chief  of  Staff's  goal  of  cleaning  up  those  sites  by  the  year 
2000,  the  Air  Force  will  need  to  spend  between  $650  million 
and  $700  million  a  year.  The  sites  that  pose  the  greatest 
hazard  to  human  health  or  the  environment  will  be  given 
first  priority.  (West,  1991:8) 

To  achieve  these  ambitious  goals,  the  Air  Force  will 
need  to  carefully  rank  their  projects  to  ensure  the  ones 
sent  forward  meet  DoD  objectives  and  can  compete 
successfully  with  other  services  for  cleanup  funds.  The 
competition  will  be  fierce.  DoD  has  approximately  18,741 
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sites  that  are  expected  to  cost  $25  billion  to  clean  up 
(Hushon,  1993). 

In  addition  to  the  year  2000  goal,  the  Air  Force  has 
established  several  other  goals  aimed  at  improving  the 
program.  First,  the  Air  Force  would  like  each  installation 
to  have  a  single  contractor  or  service  center  managing  their 
program.  That  contractor/center  should  then  be  challenged 
to  reduce  documentation  and  streamline  procedures. 
Additionally,  interim  remedial  actions  should  be  used  to  the 
fullest  extent  possible,  and  cleanups  should  be  performed  in 
phases,  when  appropriate,  to  match  the  strengths  of 
different  remedial  technologies  to  the  various  stages  of 
contaminant  removal.  Finally,  incentive-based  contracting 
methods  that  are  sensitive  to  schedules  should  be 
emphasized.  (Air  Force,  1992:8) 

DoD's  policies  affecting  the  Air  Force  are  similar  to 
those  found  in  the  NCP.  Specifically,  DoD  agencies  are 
directed  to: 

1)  Protect  the  health  and  welfare  of  installation 
personnel  and  the  public  by  dealing  with  the  contamination 
in  a  timely  manner. 

2)  Comply  with  all  applicable  requirements  for  the 
program. 

3)  Keep  regulatory  agencies  apprised  of  their 
activities  and  solicit  appropriate  comments  on  their  plans. 
DoD  will  also  negotiate  FFAs  with  EPA  and  will  follow  the 
model  FFA  language  where  possible. 

4)  Encourage  Defense  and  State  Memoranda  of  Agreement 

( DSMOA )  to  expedite  cleanup  and  reimburse  states  for  their 
technical  support  at  our  sites. 
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5)  Involve  the  public  in  DERP  activities. 

6)  Take  immediate  actions  to  address  imminent  threats. 

7)  Have  a  bias  towards  remediation  activities  instead 
of  studies. 

8)  Be  required  to  clean  up  DoD  property  before  transfer 
or  sale. 

9)  Initiate  waste  minimization  measures  to  prevent 
future  restoration  problems  (although  DERP  funding  is  not 
likely  to  be  available  for  these  initiatives) . 

10)  Comply  with  the  memorandum  of  agreement  with  the 
Agency  for  Toxic  Substances  and  Disease  Registry  ( ATSDR) . 

11)  Conduct  research  and  development  for  innovative 
technologies.  (Air  Force,  1992:3-4) 

These  priorities  are  stated  more  succinctly  in  DoD's  1991 
report  to  Congress.  In  this  report,  DoD  states  that  it 
gives  highest  priority  to  sites  that  represent  the  greatest 
risk.  Removal  of  imminent  threats,  interim  and 
stabilization  measures  that  prevent  further  deterioration 
and  achieve  life-cycle  cost  savings,  and  actions  necessary 
to  comply  with  SARA  are  also  listed  as  priorities.  (DoD, 
1992:3) 

The  Deputy  Assistant  Secretary  of  Defense  for  the 
Environment  has  overall  responsibility  for  managing  DERP. 

His  or  her  responsibilities  include  developing  and  defending 
the  budget,  providing  policy  and  guidance,  determining 
eligibility  criteria  and  priorities,  overseeing  the  program, 
and  acting  as  liaison  to  outside  agencies  (Air  Force, 
1992:5).  While  new  guidance  may  be  forthcoming,  the  former 
Secretary,  Mr.  Baca,  stressed  the  need  for  cleaning  up  sites 
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quickly,  streamlining  the  cleanup  process  to  the  extent  the 
law  allows,  and  developing  standard  procedures  to  deal  with 
similar  types  of  contamination.  (Over  80%  of  DoD's  wastes 
are  petroleum  and  solvent  products.)  (Baca,  1992:51,52). 

Many  projects  are  eligible  for  the  IRP,  but  only  about 
50%  of  those  submitted  are  projected  to  be  funded  over  the 
next  several  years  (Myers,  1991)  .  This  percentage  gets 
smaller  as  more  projects  are  identified.  Funding  for  the 
IRP  is  divided  into  three  categories.  Priority  1  projects 
are  likely  to  be  at  least  partially  funded  and  include  those 
projects  which: 

1)  Eliminate  or  reduce  human  exposure  to  contamination. 
Two  examples  of  this  type  project  are  providing  alternate 
water  supplies  and  removal  actions. 

2)  Are  time-critical.  Projects  which  are  needed  to 
meet  time-sensitive  regulatory  mandates  and  court  orders 
would  fall  into  this  category. 

3)  Maintain  compliance  with  legal  obligations  or 
official  agreements.  A  project  agreed  to  under  an  FFA  would 
meet  this  criterion. 

4)  Maintain  progress  at  proposed  or  listed  National 
Priority  List  (NPL)  sites.  Operational  support  of  a 
remediation  system  at  an  NPL  site  would  qualify  as  a 
Priority  1  project. 

Priority  1  projects  requiring  current  year  funds  are  further 
categorized  as  1A  projects,  and  those  that  could  be  funded 
in  a  subsequent  fiscal  year  are  denoted  as  priority  IB 
projects . 

The  second  category  for  funding  is  designated  Priority 
2.  Projects  which  help  ensure  adequate  health  protection 
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and  those  which  maintain  IRP  progress  (as  opposed  to  NPL 
progress)  would  be  put  into  this  category. 

The  last  category,  Priority  3  funded  projects,  are 
those  that  support  generalized,  high-potential  research  and 
development,  improve  environmental  quality,  or  demonstrate 
environmental  leadership.  Under  current  funding 
projections,  these  projects  are  unlikely  to  receive  any 
support.  (Air  Force,  1992:8-11) 

The  final  criterion  documented  in  the  programming 
guidance  to  be  used  for  ranking  DERP  project  alternatives  is 
the  Defense  Priority  Model  (DPM)  score.  DPM  was  developed 
as  a  prioritization  tool,  and  a  DPM  score  is  required  for 
most  sites  that  are  ready  for  remedial  action.  The  score 
reflects  the  potential  risk  at  a  site  and  is  a  compilation 
of  risks  for  all  likely  contaminant  transport  pathways  and 
receptors  (The  Earth  Technology  Corporation,  1992:1).  Sites 
that  score  high  on  the  DPM  are  considered  to  be  a  greater 
threat  to  human  health  or  the  environment  and  are  given 
funding  priority.  However,  mission  impact,  community 
concerns,  regulatory  considerations,  and  other  issues  can 
cause  the  DPM  to  be  overruled.  (Air  Force,  1992:16-17) 

Restoration  Methods .  The  selection  of  an  appropriate 
remediation  alternative  is  accomplished  at  the  end  of  the 
feasibility  study  stage  of  the  restoration  process.  The 
decision  depends  on  the  cleanup  goals  of  the  site,  which,  in 
turn,  are  based  on  the  ARARs  for  the  contaminants  found  at 
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the  site  (Preslo,  1989:2-4).  The  geology  of  the  site,  the 
fate  and  transport  of  the  contaminants,  background 
contaminant  levels,  and  the  regulatory  issues  and  concerns 
also  influence  the  final  decision  (Long,  1992:9;  Splitstone, 
1991:90).  A  thorough  knowledge  of  remediation  alternatives 
and  their  relative  feasibility  and  costs  are  essential  to 
the  selection  of  the  appropriate  technique  (Preslo,  1989:2- 
4)  . 

While  a  complete  description  of  the  wide  range  of 
remedial  actions  available  to  clean  up  a  site  is  beyond  the 
scope  of  this  research,  it  is  important  to  highlight  several 
key  characteristics  which  could  influence  the  allocation  of 
funds  among  the  various  alternatives.  The  characteristics 
of  interest  include  relative  costs,  uncertainty,  permanence, 
acceptability,  and  similar  criteria. 

To  begin  the  discussion  of  potential  means  of 
remediating  a  site,  it  should  be  noted  that  the  objective  of 
the  treatment  can  include:  removing  the  hazard,  blocking  the 
pathway  between  the  hazard  and  the  potential  receivers, 
relocating  or  isolating  the  receiver  from  the  hazard,  and/or 
otherwise  blocking  the  potential  for  exposure  to  the  hazard. 
Any  or  all  of  these  objectives  can  be  targeted  in  a 
treatment  methodology. 

Cleanup  techniques  are  initially  classified  on  the 
basis  of  the  location  of  the  treatment.  In  situ 
alternatives  treat  the  contaminant  in  place,  which  can  be 
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either  in  the  soil,  the  groundwater,  or  both  (Preslo, 
1989:1).  Non-in  situ  treatments  require  excavation, 
extraction,  or  some  physical  process  to  remove  the 
contaminant  and  the  medium  for  further  processing.  The 
advantages  to  leaving  the  contaminant  in  plav-e  are  the 
elimination  of  costs  for  moving  the  contaminant  and  the 
surrounding  soil  or  water,  reduced  worker  exposure  to  the 
contaminant,  and  the  potential  for  more  complete  removal  of 
the  contaminant  (Masters,  1991:260)  Non-in  situ  methods 
have  the  potential  of  reducing  the  uncertainty  of  the 
process  and  can  lead  to  faster  cleanups. 

Treatment  technologies  are  also  commonly  categorized 
being  physical,  chemical,  biological,  or  thermal  in  nature. 

A  fifth  category,  fixation/stabilization,  is  also  sometimes 
recognized.  Physical  methods  include  gravity  or  mechanical 
separation,  adsorption,  phase  change  systems,  or  filtering. 
Chemical  treatment  changes  the  contaminant  into  a  less 
hazardous  substance  through  neutralization,  oxidation, 
reduction,  or  precipitation.  Biological  methods  are 
receiving  a  great  deal  of  attention  recently.  They  all  have 
in  common  the  use  of  microorganisms  to  degrade  the 
contaminant.  Thermal  destruction  is  one  of  the  preferred 
methods  for  treating  hazardous  wastes  because  of  its 
permanence  and  speed.  Finally,  some  wastes  are  immobilized 
through  the  fixation/stabilization  process.  (Masters, 
1991:252-253) 
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Biological,  chemical,  and  thermal  processes  all  have 
the  potential  to  destroy  the  toxicity  of  the  contaminant. 
Bioreactors,  bioremediation,  composting,  and  land  farming 
all  can  transform  hazardous  organic  compounds  into  harmless 
cell  matter,  carbon  dioxide,  water,  and  various  inert 
materials  given  the  right  environmental  conditions  (Long, 
1992:24).  These  processes  can  also  be  combined  with  other 
physical  or  chemical  processes  to  enhance  the  reliability  or 
effectiveness  of  the  overall  treatment  (Long,  1992:24). 
However,  this  technology  has  only  been  found  to  be  effective 
for  a  limited  number  of  contaminants  in  limited  field 
applications  (Fox,  1991:293-303). 

Chemical  treatment  technologies  are  seldom  used  for  Air 
Force  remediation  work.  The  costs  can  be  high  compared  to 
other  methods,  and  a  high  degree  of  control  is  required 
(Long,  1992:36-37). 

Physical  methods  must  be  combined  with  other  treatment 
technologies  to  achieve  contaminant  destruction.  Air 
stripping,  carbon  adsorption,  oil-water  separation,  soil 
vapor  extraction,  and  soil  washing  are  typical  of  the 
physical  processes.  Each  of  these  methods  frees  or 
concentrates  the  waste  of  interest  for  further  processing. 
(Long,  1992:18-23) 

Given  the  legislative  bias  towards  permanence  and 
toxicity  reduction,  thermal  treatment  is  the  technology  of 
choice  for  many  contaminants.  Incineration  can  completely 
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detoxify  a  waste  and  is  useful  for  concentrating  those  that 
it  can't  (Masters,  1991:261).  Thermal  desorption  is 
typically  employed  in  combination  with  a  carbon  adsorption 
exhaust  filter  to  volatilize  and  capture  hydrocarbon 
contaminants.  It  tends  to  be  less  expensive  than 
incineration  and  easier  to  permit  (Long,  1992:35-36). 

Fixation  and  stabilization  techniques  may  combine 
chemical  and  physical  processes  to  eliminate  the  transport 
mechanism  and  thereby  negate  risk.  Asphalt  and  concrete 
incorporation  methods  have  been  tried,  but  regulators  are 
not  convinced  of  the  permanency  of  these  treatment  methods. 
Vitrification  is  also  sometimes  used  to  treat  contaminant:: . 
but  it  is  generally  too  expensive  for  most  large  scale 
applications . 

While  it  is  difficult  to  sort  through  the  myriad  of 
remediation  alternatives,  decision  support  systems  are 
emerging  to  make  the  task  easier.  One  such  tool,  the 
Remedial  Action  Cost  Engineering  and  Requirements  (RACER) 
system  under  development  at  the  Air  Force  Civil  Engineering 
Support  Agency,  will  help  ensure  the  best  technologies  are 
chosen  and  that  they  are  accurately  priced.  The  expert 
system  component  of  RACER,  the  Remedial  Action  Assessment 
System  (RAAS) ,  is  being  designed  to  select  among  available 
remediation  methods  and  combinations  of  methods  given 
available  site  data,  ARARs  are  built  into  the  system 
database,  and  the  outcome  will  be  fully  documented  with  the 
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site  data,  the  calculations  used  to  determine  the 
effectiveness  of  the  alternatives,  and  a  sensitivity 
analysis  of  the  failure  potentials.  The  parametric  cost 
estimates  provided  by  the  system  will  help  with  budgets  for 
all  aspects  of  the  remedial  process,  are  used  by  the  system 
for  a  life-cycle  cost  analysis  of  each  alternative,  and  will 
provide  a  means  to  record  historic  costs.  (Gregory, 
1992:32-33)  Another  decision  aid  is  the  Air  Force  Center 
for  Environmental  Excellence  ( AFCEE)  Remediation  Matrix- 
Hierarchy  of  Preferred  Alternatives.  This  matrix  (Appendix 
B)  ranks  technologies  in  order  of  preference  and  is  used  to 
screen  technologies  for  use  at  Air  Force  sites  (Owendoff, 
1993a)  . 

Prioritization  Modeling 

According  to  Janssen,  Nijkamp,  and  Voogd,  a 
multicriteria  environmental  analysis  should  start  with  the 
problem  definition.  From  there,  it  should  proceed  to 
identifying  pertinent  objectives  or  characteristics  within 
the  broad  range  of  environmental  criteria.  Discriminating 
characteristics  can  then  be  established  based  on  these 
objectives.  Only  after  this  prior  step  should  an 
appropriate  evaluation  method  or  model  for  the  DSS  be 
chosen.  Once  the  model  is  selected,  one  can  evaluate  the 
problem  and  test  the  model.  (Janssen,  1985:1,2) 
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Decision  Criteria  and  Models.  Because  of  the  multitude 
and  complexities  of  discriminating  criteria  for  this 
resource  allocation  problem,  some  means  of  achieving  a 
weighted  hierarchy  of  these  criteria  will  be  established  in 
the  research.  The  outcome  should  be  supported  by  key  DERP 
managers  to  instill  confidence  in  the  resulting  model.  The 
Delphi  technique  is  one  possible  method  for  achieving  these 
results . 

The  Delphi  technique  takes  advantage  of  the  benefits  of 
group  thinking  while  eliminating  many  of  the  disadvantages. 
The  first  step  involves  mailing  questionnaires  to  key 
decision  makers.  Their  responses  are  summarized,  and  the 
summaries  are  sent  back  with  a  request  to  reevaluate  and 
comment  on  the  summary.  Several  iterations  of  this  process 
may  be  accomplished  before  the  surveyor  is  satisfied  that  a 
consensus  is  reached.  (Oxenfeldt,  1979:146) 

It  is  important  to  distinguish  between  first-order  and 
second-order  selection  criteria  before  finishing  the  Delphi 
survey.  First-order  criteria  are  those  conditions  that  must 
be  met.  Second-order  criteria  should  be  met  but  aren't  as 
critical.  (Janssen,  1985:352-353)  Oxenfeldt  calls  these 
ultimate  and  instrumental  criteria,  and  he  states  that  the 
criteria  in  each  class  can  be  either  compatible  or 
conflicting.  When  the  criteria  will  be  used  to  determine  an 
allocation  of  limited  resources  (as  in  this  research 
problem) ,  Oxenfeldt  notes  the  importance  of  establishing  a 
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ranking  among  the  criteria.  He  cautions,  however,  that  the 
ranking  will  vary  depending  on  the  circumstances  and 
external  influences  prevailing  at  the  time  the  priorities 
are  set.  (Oxenfeldt,  1979:185) 

It  is  important  to  note  that  one  of  the  primary  reasons 
for  developing  a  decision  model  is  to  force  managers  to  be 
objective  and  obtain  information  that  protects  them  from 
making  decisions  by  inpulse  and  emotion  (Oxenfeldt, 
1979:139).  The  success  of  the  model  will  be  measured  by  how 
well  the  Air  Force  achieves  the  stated  objectives  (Chankong 
and  Haimes,  1983:8). 

The  Delphi  technique,  the  surveys  of  experts,  and  MCDMs 
will  be  discussed  in  further  detail  in  the  methodology 
section  (Chapter  Three) . 

Risk  as  a  Decision  Parameter.  The  use  of  risk  as  a 

decision  parameter  is  critical  to  meeting  legislative 

mandates.  As  stated  earlier,  the  main  goals  of  CERCLA  and 

DERP  are  to  make  sites  cleaner  and  safer.  Risk  is  the 

primary  method  used  to  measure  the  relative  safety  of  a 

site.  As  the  Science  Advisory  Board  (SAB)  notes: 

One  tool  that  can  help  foster  the  evolution  of  an 
integrated  and  targeted  national  environmental  policy 
is  the  concept  of  environmental  risk.  ...  The  concept 
of  environmental  risk,  together  with  its  related 
terminology  and  analytical  methodologies,  helps  people 
discuss  disparate  environmental  problems  with  a  common 
language.  It  allows  many  environmental  problems  to  be 
measured  and  compared  in  common  terms,  and  it  allows 
different  risk  reduction  options  to  be  evaluated  from  a 
common  basis.  Thus  the  concept  of  environmental  risk 
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can  help  the  nation  develop  environmental  policies  in  a 
consistent  and  systematic  way.  (EPA,  1990b: 2) 

In  the  same  report,  the  SAB  also  recommended  the  EPA  use 

risk  reduction  in  setting  priorities  for  environmental 

protection.  (EPA,  1990b: 16). 

Risk,  however,  can  have  many  interpretations  depending 

upon  its  use  and  context.  Additionally,  the  measurement  of 

risk  for  use  in  remedial  actions  is  fraught  with  uncertainty 

and  controversy.  (Chiras,  1991:327-328) 

The  technical  definition  of  environmental  risk  is  the 

probability  of  an  adverse  response  to  a  specific  dose  over  a 

specific  period  of  time  (Barill,  1991).  This  definition  is 

based  upon  the  knowledge  that  any  substance  has  the 

potential  to  cause  an  adverse  response,  given  that  it  is 

received  in  a  large  enough  dose,  or  over  a  long  enough 

period  of  time.  A  second  definition  used  is  the  legal  or 

regulatory  definition.  It  states  that  risk  is  the  potential 

to  experience  harm  from  a  hazard,  or  the  possibility  of 

human  exposure  to  a  hazard  (Barill,  1991) . 

We  measure  risk  in  terms  of  the  probability  of  harm  or 

exposure.  Several  factors  give  rise  to  this  probabilistic 

nature  of  risk: 

1)  Variables  are  not  known  and  cannot  be  predicted  with 
certainty. 

2)  The  science  of  risk  is  not  fully  understood. 

3)  The  processes  involved  are  probabilistic  and  their 
relationships  are  complex. 
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4)  The  uncertainty  often  depends  on  the  perspective  of 
the  individual  at  risk.  (Morgan,  1981:61) 

In  the  following  sections,  the  process  for  determining 
risk  and  specific  factors  that  give  rise  to  uncertainty  are 
examined . 

Risk  Assessment.  Environmental  protection  based 
on  the  concept  of  risk  begins  with  the  risk  assessment. 

Risk  assessment  is  the  process  by  which  the  sources  of 
hazards,  the  exposure  routes,  and  the  dose-response 
relationships  are  assessed  and  the  risks  are  characterized 
(Cohrssen,  1989:7-8).  This  process  is  paramount  to  decision 
making  and  provides  the  information  needed  for  assessing 
specific  control  options,  setting  priorities,  allocating 
resources,  and  assessing  damages  for  compensation  purposes 
(Hattis  and  Smith,  1987:60). 

The  first  step  in  a  risk  assessment  is  to  determine  the 
sources,  amounts,  and  duration  of  exposures  to  toxic  agents. 
For  DERP  work,  the  sources  are  a  result  of  past  hazardous 
waste  disposal  and  spills.  Several  distinct  sites  may 
contribute  to  the  overall  risk,  and  each  source  may  contain 
numerous  hazardous  chemicals.  Depending  on  site  specifics 
such  as  the  geology  of  the  site  and  the  nature  of  the 
contaminant,  the  amount  and  duration  of  the  exposure  will 
vary . 

Exposure,  as  used  above,  is  the  term  for  plant,  animal, 
and  human  contact  with  harmful  chemicals.  For  humans,  the 
most  common  routes  of  exposure  are  ingestion  and  inhalation. 
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The  primary  media  through  which  chemical  exposures  occur  are 
air,  water,  soil,  and  dust.  Pathways  describe  the  means  and 
media  by  which  the  exposures  occur,  and  several  pathways  may 
be  involved  for  each  exposure.  For  instance,  a  chemical  may 
be  transmitted  through  air,  deposited  in  soil,  dissolved  in 
water,  absorbed  by  other  plants  or  animals,  etc.  before  a 
human  is  ever  exposed. 

Both  the  source  and  the  exposure  assessments  introduce 
uncertainty  in  the  overall  risk  assessment.  The  choices 
made  regarding  sampling  locations  and  contaminants  to 
sample,  the  randomness  of  climatic  conditions,  assumptions 
about  the  exposure  pathways,  and  the  heterogeneity  of  the 
site  all  can  introduce  uncertainty  into  the  calculation  of 
risk . 

Another  critical  step  in  a  risk  assessment  is  the  dose- 
response  assessment  to  determine  the  relationship  between 
the  chemical  exposure  and  the  resulting  response  by  the 
recipient  of  that  chemical.  The  toxic  agents  at  a  site 
generally  fall  into  one  of  three  categories,  each  with  a 
unique  dose-response  relationship.  First  are  the  fast 
poisons.  These  are  chemicals  which  cause  acute  symptoms  but 
do  not  persist.  That  is,  if  the  toxin  is  ingested  or 
otherwise  taken  into  the  body,  it  does  not  linger. 

Ingestion  of  fast  poisons  generally  does  not  occur  on  a 
regular  or  ongoing  basis.  (Rodricks,  1992:49) 
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The  second  category  is  the  slow  poisons.  Slow  poisons 
are  chemicals  which  are  normally  ingested  over  longer 
periods  of  time.  They  may  either  accumulate  in  the  body, 
causing  damage  only  after  reaching  a  certain  concentration, 
or  they  may  be  ingested  on  a  regular  basis,  causing  a  small 
amount  of  damage  with  each  event.  In  the  latter  case,  the 
damage  is  not  usually  noticed  until  the  process  has  gone  on 
for  quite  some  time.  (Rodricks,  1992:66) 

The  third  group  of  toxic  agents  is  the  mutagens,  which 
include  carcinogens  and  teratogens.  Mutagens  include  the 
general  class  of  chemicals  which  alter  the  genetic  structure 
of  affected  cells  in  the  subject.  Carcinogens  are  chemicals 
which  specifically  cause  cancer,  and  teratogens  are 
chemicals  which  specifically  cause  birth  defects  (Masters, 
1991:196).  For  substances  that  induce  a  carcinogenic 
response,  it  is  generally  (and  conservatively)  assumed  that 
there  is  no  'no-effect'  dose,  meaning  that  any  dose  of  the 
carcinogen  will  create  some  likelihood  of  cancer  (Masters, 
1991:201) . 

Using  the  dose-response  data,  it  is  possible  to 
determine  if  there  exists  a  threshold  of  toxicity  for  that 
chemical.  In  his  book,  Calculated  Risks.  J.V.  Rodricks 
notes : 

If  for  every  chemical  in  the  environment  we  knew  the 
range  of  'no-effect'  doses  and  the  point  at  which 
toxicity  begins  to  appear  -  the  point  at  which  the 
threshold  of  toxicity  is  passed  -  we  could  then  act  to 
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prevent  exposures  from  ever  reaching  the  levels  at 
which  harmful  doses  are  created.  (Rodricks,  1992:158) 

The  uncertainty  of  the  effects  of  the  toxic  agent  in 

the  dose-response  relationship  is  often  related  to  the 

length  of  the  exposure  period  and  the  complexity  of  the 

damage  mechanisms.  The  dose-response  relationships  of 

carcinogens  involve  the  most  uncertainty.  Dose-response 

information  is  generally  gathered  through  animal  testing. 

Test  animals  are  given  extremely  high  doses  of  chemicals  for 

extended  periods  of  time  in  an  attempt  to  accelerate  the 

effects  of  exposure.  These  doses  and  exposure  durations  are 

then  extrapolated  back  to  obtain  theoretical  responses  for 

lower  levels  of  dosages  and  exposures  in  humans.  The 

general izability  of  these  tests  are  the  source  of  much 

controversy.  There  is  no  way  to  be  certain  that 

extrapolation  back  to  the  levels  to  which  humans  are  likely 

to  be  exposed  is  valid.  Even  with  extremely  large  numbers 

of  animals  in  a  bioassay,  the  lowest  risks  that  can  be 

calculated  exceed  the  acceptable  risk  ranges  mandated  for 

humans  by  several  orders  of  magnitude.  (Masters,  1991:202) 

Knowledge  of  the  sources,  the  many  exposure  pathways, 

and  the  dose-response  relationships  are  critical  to  risk 

characterization.  Risk  characterization  is  the  analysis 

phase  whereby  an  overall  numerical  estimate  or  probability 

of  an  individual  being  affected  by  some  hazard  is 

calculated,  the  uncertainty  involved  is  characterized,  and 

ways  to  reduce  the  risk  are  determined.  (Cohrssen,  1989:84) 
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Risk  characterization  has  also  been  described  as 
assessing  the  worst-case  impacts  via  a  variety  of 
conservative  input  assumptions  and/or  analytical  techniques 
(Tusa,  1992:46).  As  a  result,  these  assumptions  frequently 
overstate  the  risks  of  hazards  by  a  factor  of  50  or  more 
(Lave  and  Grunspecht,  1991:685)  .  Given  this  uncertainty  and 
the  limitations  inherent  in  the  process,  the  risk  estimates 
should  be  qualified  with  a  description  and  discussion  of  the 
uncertainties  and  analytic  assumptions  (Cohrssen,  1989:84). 

Risk  Management .  Risk  management  uses  the 
information  from  a  risk  assessment  in  an  attempt  to  reduce 
that  risk.  It  describes  the  process  whereby  decisions  are 
made  about  whether  an  assessed  risk  needs  to  be  reduced  and 
on  the  means  that  should  be  used  to  achieve  the  desired 
reduction  (Rodricks,  1992:182).  In  addition  to  protecting 
human  health,  the  risk  management  actions  should  be 
protective  of  the  environment,  be  consistent  with  relevant 
laws  and  regulations,  and  consider  social,  economic,  and 
political  values  (Ruckelshaus,  1985:25). 

Risk  reduction  in  the  context  of  DERP  refers  to  removal 
actions  or  remedial  actions  taken  to  'clean  up'  a  site, 
thereby  lowering  the  potential  for  harm  to  the  likely 
recipients.  Completely  eliminating  a  hazard  is  never  cost 
feasible  and  is  rarely  technically  feasible.  The  decision 
to  be  made  is  where  to  establish  the  trade-off  between  risk 
reduction,  remediation  cost,  and  other  criteria.  Current 
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remediation  decisions  appear  :o  be  driven  largely  by  cost, 

policy,  ARARs,  and  subjective  judgment  rather  than  current 

or  future  risk  levels.  (Travis  and  Doty,  1989:1333) 

Figure  1  shows  a  summary  of  the  risk  management 

process,  given  the  information  from  the  risk  assessment 

procedures  referred  to  previously. 

Secondary  Decision  Criteria.  In  their  report  on  risk 

assessment,  Doty  and  Travis  note  that: 

Reducing  public  health  risk  involves  not  only  defining 
risk,  but  also  involves  using  risk  assessment  as  a 
priority  setting  tool  and  defining  risk  reduction 
within  the  confines  of  the  current  state  of  technology 
and  the  resources  available  with  which  to  address  risk. 

Here,  Doty  and  Travis  indicate  that  the  risk  assessment 

should  be  used  in  combination  with  several  other  criteria  in 

setting  priorities  for  risk  reduction  (Doty  and  Travis, 

1990:2).  Among  those  mentioned  in  the  literature,  cost- 

effectiveness,  technical  and  contractual  feasibility, 

innovation  value,  and  community  input  appear  to  be  the  most 

important.  The  results  of  this  research  will  determine  how 

much  weight  each  of  these  criteria  is  to  be  given  in  the 

resulting  MCDM  and  if  additional  factors  should  be 

considered  in  the  final  decision  model. 

Cost .  Cost-effectiveness  was  mentioned 
consistently  in  the  legislation,  implement ing  guidance,  and 
DoD  policy  along  with  risk  reduction  as  an  important 
criterion  for  establishing  priorities.  The  Superfund 
Handbook  defines  a  cost-effective  alternative  as  the  method 
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Figure  1.  Risk  Management  Summary  (Tusa,  1992:48). 


identified  as  the  best  available  in  terms  of  reliability, 
permanence,  and  economics.  It  notes  that  the  least 
expensive  method  is  not  necessarily  the  best  alternative. 
(Lucero,  1989:5) 

While  selecting  an  alternative  that  meets  the  above 
definition  would  seem  intuitive,  an  economic  analysis  should 
be  accomplished  to  focus  the  manager's  thinking,  surface 
assumptions,  and  to  document  the  methodology  and  evidence. 
Three  basic  principles  should  be  followed  in  the  analysis. 
First,  all  reasonable  means  of  fulfilling  the  program 
objectives  should  be  evaluated.  Second,  the  alternatives 
should  be  analyzed  in  terms  of  their  life-cycle  funding 
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implications  and  their  life-cycle  costs/benefits.  Last,  the 
time  value  of  money  must  be  considered.  (Research  and 
Education  Association,  1982:3) 

The  steps  to  an  economic  analysis  follow  the  standard 
problem-solving  format.  First,  the  objectives  must  be 
defined.  Next,  alternatives  are  generated  and  assumptions 
are  listed.  The  costs  and  benefits  of  each  alternative  are 
then  determined  and  compared  so  that  a  hierarchy  can  be 
established.  Finally,  because  uncertainties  exist, 
sensitivity  analysis  should  be  performed.  (Research  and 
Education  Association,  1982:4-7) 

The  difficult  step  in  this  process  is  quantifying  the 
costs  and  benefits.  Since  many  of  the  costs  and  benefits  in 
environmental  problems  are  externalities  and  are  not  valued 
in  the  marketplace,  little  relevant  data  is  available,  and 
both  are  highly  uncertain.  Additionally,  previous  attempts 
at  placing  a  value  on  public  health  and  environmental 
improvements  have  been  highly  controversial.  (Lave, 
1991:682-683) 

Feasibility.  Preslo  and  his  associates  argue  that 
both  relative  cost  and  feasibility  influence  the  choice  in 
an  environmental  problem.  They  further  break  the  category 
of  feasibility  into  core  areas,  which  include  both  technical 
and  implementational  feasibility.  (Preslo,  1989:2-4) 

Technical  feasibility  addresses  the  effectiveness  of 
the  alternative  given  a  wide  range  of  variables.  For 
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example,  soil  chemistry,  porosity,  permeability,  adsorption 
characteristics,  and  moisture  content  all  can  affect  the 
suitability  of  the  treatment  methods  for  various 
contaminants.  Site-specific  climate  conditions  such  as 
temperature,  wind,  humidity,  and  average  precipitation  also 
influence  the  effectiveness  of  the  alternatives.  (Preslo, 
1989:20-23)  Therefore,  any  decision  model  will  have  to 
consider  the  inherent  uncertainties  and  site-specific  needs 
of  each  alternative. 

Implementation  feasibility  addresses  the  ability  to 
design  the  alternative,  to  obtain  the  necessary  equipment  to 
install  or  operate  the  system,  to  treat  and  dispose  of  the 
wastes,  and  to  monitor  and  obtain  a  regulatory  permit  for 
the  system  (Preslo,  1989:2-4)  .  The  Air  Force  also  has  to 
examine  the  procurement  feasibility  for  first  obtaining  a 
flexible  enough  contract  for  environmental  work  and  then 
retaining  options  on  that  contract. 

Innovative  Value.  With  the  staggering  cleanup 
costs  facing  the  Nation  today,  and  with  the  relative 
inexperience  of  the  remediation  industry,  a  great  deal  of 
emphasis  is  being  placed  on  developing  innovative 
technologies.  These  new  technologies  hold  the  promise  of 
quicker,  less  expensive,  more  permanent  treatment  of  the 
toxic  waste  problem,  but  at  the  same  time  introduce  a  great 
deal  of  uncertainty.  While  the  potential  for  improvement  is 
great,  the  value  that  should  be  placed  on  developing 
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innovative  technologies  is  difficult  to  quantify.  In  turn, 
the  value  that  should  be  placed  on  this  criterion  in  a 
prioritization  model  will  also  be  difficult  to  determine. 

Other  Criteria.  As  noted  in  the  previous 
discussions  of  objectives,  several  other  criteria  should  be 
considered  in  the  final  MCDM.  Possibilities  include: 

1)  Public  involvement  in  deciding  the  overall  priority 
of  a  site.  The  public  ultimately  pays  for  the  cleanup  and 
are  the  ones  exposed  to  the  risk,  therefore,  their  concerns 
should  be  represented  in  the  MCDM. 

2)  Compliance  with  existing  regulatory  agreements  and 
schedules.  Base-level  commitments  were  made  to  both  the 
regulators  and  the  affected  public  that  should  enter  the 
decision.  At  the  same  time,  the  schedules  agreed  to  were 
based  more  on  technical  feasibility  than  current  economic 
realities . 

3)  Past  execution  history.  In  some  cases,  a  large 
percentage  of  prior  year  funds  have  been  obligated,  but  have 
not  been  spent  (Owendoff ,  1993a) .  Should  more  money  be 
allocated  to  those  projects  before  the  prior  year  funds  are 
expended? 

4)  The  effectiveness  in  allocating  past  funds.  This 
might  be  an  important  predictor  of  future  performance.  In  a 
recent  article,  it  was  estimated  that  as  much  as  70-75 
percent  of  the  cost  of  cleaning  up  Department  of  Energy 
(DOE)  sites  is  wasted.  The  average  efficiency  was  thought 
to  be  in  the  range  of  60-65  percent.  Additionally,  huge 
disparities  existed  in  the  price  of  services  among  various 
agencies  that  could  not  be  accounted  for  by  geographic  or 
technological  differences.  Prioritizing  projects  based  on 
past  effectiveness  could  be  an  incentive  to  improve 
performance.  (Pasternak,  1992:34-36) 

Measuring  these  criteria  for  use  in  the  MCDM  and 
establishing  a  relative  weight  may  prove  difficult.  One 
possibility  would  be  to  lump  these  kinds  of  criteria  into  a 
single  metric  and  let  the  MAJCOMs  determine  their  relative 
importance.  MAJCOMs  already  rank  their  projects  and  submit 
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to  the  Air  Force  only  those  that  meet  Air  Staff  standards, 
can  successfully  compete  for  funding,  and  are  known  to  be 
important  to  the  various  Installation  Commanders  or  the 
affected  public.  They  also  provide  an  important  check  on 
the  validity  of  the  installation's  submission. 

The  benefits  and  shortcomings  of  using  MAJCOM 
priorities  is  comparable  to  that  placed  on  any  group 
decision  making.  Some  of  the  benefits  of  this  type  decision 
making  might  include  a  better  consideration  of  the  problem 
as  a  whole.  Groups  tend  to  consider  more  facets,  have  more 
background  knowledge,  and  can  reduce  the  probability  of 
missing  a  key  aspect  of  the  problem.  Additionally,  allowing 
this  input  might  instill  a  commitment  on  behalf  of  the 
participants  in  the  program  and  the  resulting  decisions. 
Finally,  the  process  could  better  facilitate  the 
communication  of  both  the  decisions  and  the  rationale  for 
the  decisions  to  impacted  parties.  (Kroenke,  1992:160-161) 

Findings,  From  Related,,  Analyses 

A  number  of  other  reports  have  been  produced  for  the 
DoD,  individual  DoD  services,  and  the  EPA  in  the  last  three 
years  which  have  addressed  the  issues  involved  in  priority¬ 
setting  with  limited  funding  capability.  The  following 
sections  give  a  brief  overview  of  these  publications. 

The  Keystone  Report.  The  most  recent  report  on 
priority-setting  in  the  IRP,  produced  by  The  Keystone 
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Center,  promulgates  a  set  of  recommendations  for  improving 
the  federal  priority-setting  process.  These  recommendations 
include: 

1)  Improving  the  dissemination  of  Federal  Facilities 
Environmental  Restoration  (FFER)  related  information, 

2)  Improving  stakeholder  involvement  in  key  FFER 
decisions,  with  special  emphasis  on  the  use  of  site-specific 
advisory  boards  (SSABs),  and 

3)  Improving  consultation  on  FFER  funding  decisions  and 
setting  priorities  in  the  event  of  funding  shortfalls. 
(Keystone,  1993 :v) 

The  Keystone  report  emphasizes  inproving  the  role  of, 
and  increasing  the  involvement  of  the  affected  public  in  the 
decision  making  process.  The  report  delineates  procedures 
for  stakeholder  involvement  through  SSABs,  including 
determining  when  an  SSAB  is  required,  determining  SSAB 
membership,  and  appropriate  operating  procedures  for  the 
SSAB.  The  report  also  proposes  a  list  of  the  major 
functions  of  the  SSAB,  and  the  decision  milestones  in  ,*hich 
the  SSAB  should  be  actively  involved. 

The  basic  recommendation  of  the  Keystone  Report  is  that 
a  policy  of  'fair  share  allocation’  be  adopted.  This 
implies  that  if  Congress  appropriates  only  a  fraction  of  the 
funding  requirement  of  the  FFER  program,  then  all  sites 
would  share  equally  in  the  amount  of  the  funding  shortfall, 
thus  receiving  only  a  fraction  of  the  requested  funding. 
(Keystone,  1993:45)  This  policy  would  force  prioritization 
of  sites  down  to  lower  agencies  who  submit  the  funding 
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requirements.  However,  the  system  could  easily  be  'gamed' 
by  increasing  the  amount  of  funding  requested  for  each  site, 
thus  defeating  the  intent  and  practicality  of  fair  share 
allocation. 

The  recommendation  of  the  Keystone  report  follows  from 

their  opinion  of  the  insurmountable  obstacles  in  developing 

a  prioritization  system.  In  their  words: 

The  Committee  believes  that:  1) existing  data  and 
science  are  currently  inadequate  to  determine  objective 
consensus  clean-up  priorities;  2) factors  other  than 
environmental  and  human  health  risk  deserve 
consideration  in  allocating  clean-up  resources; 

3) broadly  acceptable  and  objective  methods  for 
evaluating  some  of  the  criteria  relevant  to  the 
allocation  of  clean-up  priorities  do  not  currently 
exist,  and,  in  some  cases,  may  never  exist  and  may  even 
be  inappropriate;  and  4) regardless  of  any  party's 
opinion  about  the  quality  of  available  data  and 
science,  it  is  appropriate  in  a  democracy  to  allow  a 
variety  of  affected  interests  to  provide  input  on 
decisions  that  affect  them.  (Keystone,  1993:44) 

The  Coalition  on  Suoerfund  Reports.  In  early  1992,  two 

separate  reports  were  presented  to  the  Coalition  on 

Superfund  which  address  the  emerging  issue  of  Superfund 

priorities.  The  following  sections  briefly  outline  the 

details  of  these  reports. 

The  Arthur  D.  Little  Report.  This  report  was 
subcontracted  to  the  Center  for  Technology,  Policy  and 
Industrial  Development  at  the  Massachusetts  Institute  of 
Technology,  and  was  presented  in  February  1992.  It 
maintains  that  Superfund  sites  should  be  prioritized  based 
upon  two  main  criteria:  the  urgency  of  the  clean-up  and  the 
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availability  of  resources  to  address  the  problem.  (Center 
for  Tech...,  1992:6)  These  criteria  are  further  broken  down 
into  the  following  eight  specific  system  requirements: 

1)  The  system  must  sort  sites  as  they  are  encountered. 

2)  The  system  should  not  be  wasteful. 

3)  The  system  should  be  able  to  differentiate  technical 
feasibility  among  alternatives  and  sites. 

4)  The  process  should  address  sites  in  a  timely  manner 
to  avoid  significant  worsening  of  site  contamination. 

5)  The  process  should  be  flexible,  to  allow  for  re¬ 
ordering  of  priorities. 

6)  The  system  should  have  a  means  of  deferring  sites 
with  low  urgency. 

7)  The  system  should  have  a  means  of  deferring  sites 
where  the  effectiveness  of  remedial  technologies  is 
uncertain. 

8)  The  system  should  include  the  option  to  stabilize  a 
site  to  defer  allocation  of  resources.  (Center  for  Tech..., 
1992:7) 

The  report  presents  six  recommendations  for  the 
improvement  of  the  existing  Superfund  site  prioritization 
procedures : 

1)  Site  Categorization.  Analogous  to  operable  units, 
sites  with  similar  characteristics  should  be  grouped 
together  for  prioritization. 

2)  Creation  of  a  No  Action,  Monitor  Pathway.  Deferral 
of  action  until  action  is  urgent. 

3)  Immediate  Action/No  Action,  Monitor  Pathway. 

Similar  to  recommendation  2,  except  the  decision  to  monitor 
should  follow  an  interim  risk-reducing  action. 

4)  Integration  of  Formal  HRS  Re-scoring  into 
Recommendations  2  and  3.  Sites  are  re-scored  after  the 
assessment  and  interim  actions.  Sites  below  the  trigger 
level  can  be  removed  from  NPL. 


40 


* 


5)  Creation  of  Recontrol  and  Monitor  Pathway.  Sites 
where  remediation  technologies  are  uncertain  are  stabilized 
and  deferred  until  action  is  urgent. 

6)  Broadening  of  Removal  Authority.  An  expanded  set  of 
removal  actions  should  be  allowed  and  should  be  reflected  in 
formal  HRS  scoring,  thereby  avoiding  the  need  for  inclusion 
on  the  NPL.  (Center  for  Tech...,  1992:2) 

While  the  emphasis  of  this  report  is  clearly  on 
prioritizing  sites,  it  does  not  present  a  working  decision 
support  system.  The  six  recommendations  for  modification  of 
the  Superfund  regulatory  framework  will  allow  greater 
flexibility  for  allocation  of  resources  to  sites  with 
greater  urgency,  but  they  do  not  offer  much  guidance  in 
selection  of  those  sites  whose  urgency  has  been  established. 

The  Putnam.  Haves,  and  Bartlett  Report.  This 
report  was  presented  to  the  Coalition  on  Superfund  in  March 
1992.  It  emphasizes  the  prioritization  of  individual 
actions  within  sites  but  does  not  necessarily  emphasize 
prioritizing  the  sites  themselves.  (Putnam,  et  al,  1992:2) 
The  report  approached  the  problem  from  the  perspective  that 
certain  actions  at  a  site  will  achieve  greater  risk 
reduction  at  lower  cost  than  other  actions.  Therefore,  the 
individual  actions  within  the  site  should  be  prioritized  to 
achieve  the  greatest  cost/benefit  efficiency. 

The  report  suggests  three  separate  components  be 
incorporated  into  the  Superfund  prioritization  approach: 

1)  Identification  of  early  actions  which  may  be  taken 
at  sites  that  are  quick  and  could  significantly  reduce  risk 
at  a  relatively  low  cost. 
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2)  Identification  of  possible  alternative  site 
'destinations',  such  as  innovative  remedial  technologies  or 
special  'categorical'  studies. 

3)  Use  of  a  prioritization  scheme  after  the  signing  of 
the  ROD  which  will  identify  and  prioritize  actions  similar 
to  the  early  actions  of  the  first  suggestion.  (Putnam,  et 
al,  1992:2-4) 

The  emphasis  of  this  report  is  on  identifying  those 
actions  wichin  a  Superfund  site  which  can  most  efficiently 
reduce  risk  at  the  lowest  cost.  The  report  does  not  clearly 
identify  whether,  or  how,  these  actions  should  be 
prioritized  across  sites.  Therefore,  it  does  not  contribute 
to  the  problem  of  first  deciding  which  sites  to  address 
under  the  constraint  of  a  limited  budget. 

The  Applied  Decision  Analysis  (ADA)  Report.  The  ADA 
report  was  presented  to  the  DoD  in  1990,  and  outlines  three 
conceptual  priority  systems.  These  systems  are: 

1)  A  site  ranking  system  which  ranks  DoD  sites 
according  to  urgency  of  remediation, 

2)  A  project  prioritization  system  which  ranks  projects 
according  to  the  estimated  benefit  they  would  produce  per 
dollar  spent,  and 

3)  A  'marginal  analysis'  funding  allocation  system 
which  distributes  funds  among  Services  and  installations  in 
the  order  of  greatest  marginal  benefits.  (ADA,  1990:2-6) 

To  date,  none  of  these  suggestions  have  been  explored 
further  or  implemented.  The  approach  taken  by  ADA  does  not 
lend  itself  to  widespread  use  across  organizations,  mainly 
because  there  has  been  no  consensus  on  what  the  system 
objectives  should  be.  Disagreements  precluding  the  use  of 
these  systems  have  arisen  over  1)  the  basic  unit  of 


prioritization  (e.g.,  ranking  sites  vs.  projects  vs.  actions 
vs.  requirements),  2)  the  inclusion  of  all  program 
activities  vs.  some  subset  of  program  activities,  and  3)  the 
basis  for  decisions  (e.g.,  risk-based  decisions)  (Reed, 
1993:1).  Also,  "[the  various]  Services  do  not  want  to  be 
locked  into  a  decision  support  system  -  they  want  to  retain 
ultimate  flexibility  in  allocating  resources*  (Reed, 

1993:1) . 

Clearly,  the  interest  in  this  issue  indicates  that  it 
has  widespread  impacts  and  applications  for  a  variety  of 
organizations.  However,  to  date,  no  particular  approach  to 
resource  allocation  or  site  prioritization  has  had  wide 
acceptance  either  within  its  target  organization,  or  across 
organizations . 

Concision 

The  Air  Force  and  other  branches  of  the  Department  of 
Defense  are  required  by  legislation  such  as  CERCLA  and  SARA 
to  accomplish  the  remediation  of  hazardous  waste  sites. 
Prioritizing  the  remedial  investigations,  designs,  and 
actions  for  the  4,000  Air  Force  hazardous  waste  sites 
currently  listed  in  the  DERP  program  is  a  monumental  task. 

Among  the  issues  which  must  be  considered  while  ranking 
the  projects  are  the  legislative  and  regulatory  objectives, 
DoD  program  objectives,  the  remediation  methods  chosen,  and 
the  risk  at  each  site.  Possible  decision  variables  include 


risk  reduction,  cost,  feasibility,  innovative  value,  and  a 
host  of  other  criteria.  The  decision  on  which  of  these 
criteria  to  include  in  the  final  MCDM  and  their  relative 
weights  will  depend  on  the  outcome  of  the  surveys. 

A  valid  decision  model  will  account  for  all  those 
factors  that  are  judged  to  be  pertinent.  Additionally,  it 
will  enable  the  Air  Force  to  demonstrate  its  objectivity, 
consistency,  and  stewardship  in  complying  with  this 
important  program  to  protect  human  health  and  the 
environment . 
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This  chapter  outlines  the  procedures  used  in  developing 


a  multicriteria  decision  model  (MCDM)  for  use  in  an  Air 
Force  decision  support  system  (DSS)  to  establish  a  funding 
hierarchy  among  Defense  Environmental  Restoration  Program 
(DERP)  projects.  Given  the  nature  of  the  problem  and  the 
time  constraints  for  the  research,  two  procedures  were 
simultaneously  employed.  Each  procedure  was  based  on 
different  assumptions  and  designed  to  accomplish  a  specific 
objective. 

Background 

As  the  literature  review  suggests,  there  are  numerous 
conflicting  opinions  on  the  criteria  to  be  used  for  deciding 
Installation  Restoration  Program  (IRP)  priorities  and  on 
which  criterion  is  most  important.  The  various  authors  have 
either  voiced  their  opinions,  often  with  excellent 
arguments,  without  establishing  a  specific  prioritization 
method  or  metrics  for  their  criteria,  or  have  stopped  short 
of  completing  their  models  because  of  a  lack  of  consensus 
among  the  stakeholders . 

In  contrast  to  the  earlier  work,  the  motivation  for 
this  research  fits  the  category  Morgan  termed  'substance- 
focused'  motivation.  It  is  intended  to  not  only  develop 
further  insight  and  understanding  into  the  problem  for  both 
the  decision  maker  and  other  interested  stakeholders,  but 


more  importantly,  its  focus  is  on  answering  a  formulated 
policy  question.  (Morgan,  1990:34) 

The  resulting  decision  model  is  intended  to  be  a  'first 
step',  or  as  Clark  puts  it,  an  'adaptive  design'.  The 
problem  is  complex  and  varies  from  one  context  to  the  next; 
therefore,  it  will  be  extremely  difficult  to  solve  it 
precisely.  Instead,  the  goal  will  be  to  seek  an  adaptive 
solution  that  can  be  improved  as  more  information  is 
obtained.  (Morgan,  1990:28) 

Rationale  For  Twe  .Mod, sis 

The  process  of  choosing  the  best  research  method  to 
develop  this  ‘adaptive  design*  presented  a  difficult 
challenge.  Several  studies  have  shown,  and  the  literature 
review  confirmed,  that  the  stakeholders  often  can't  agree  on 
the  problem  to  be  solved  (e.g.,  prioritizing  sites  according 
to  risk  or  projects  according  to  cost-benef it ) ,  the 
objectives  for  the  research  (e.g.,  difficulty  in 
establishing  a  consensus  on  the  goals  of  Superfund  and 
choosing  metrics  that  will  be  useful  for  a  DSS) ,  the  past 
and  future  states  of  the  environment  (e.g.,  how  clean  is 
clean) ,  or  even  the  consequences  of  alternative  courses  of 
actions  (e.g.,  does  a  particular  technology  even  work). 

This  leads  to  different  stakeholders  solving  different 
problems,  and  in  particular,  a  difficulty  in  analyzing  data 
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generated  at  different  levels  of  management.  (Vari, 
1986:317) 

These  same  difficulties  led  Goldstein  to  conclude,  as 
paraphrased  by  Morgan: 

The  process  of  choosing  decision  criteria  and  policy 
strategies  may  not  involve  a  single  decision  and  may 
not  precede  all  analysis.  Until  a  problem  has  been 
subjected  to  some  analysis,  one  may  not  understand  it 
well  enough  to  make  reasonable  choices.  Indeed,  even 
while  different  parties  are  arguing  that  a  problem 
should  be  framed  in  terms  of  different  decision 
criteria  and  policy  strategies,  it  is  sometimes 
possible  to  perform  an  analysis  that  can  accommodate 
and  allow  a  comparison  among  these  alternatives. 

(Morgan,  1990:29) 

Given  these  insightful  comments,  it  was  decided  the 
research  should  not  look  for  an  ' ideal '  methodology  but 
instead  should  utilize  the  strengths  of  two  common 
strategies  and  compare  the  results.  Both  are  briefly 
described  below  and  are  discussed  in  detail  in  the  following 
sections . 

The  first  procedure  chosen,  the  Delphi  procedure,  was 
chosen  to  overcome  the  stakeholder  agreement  dilemma.  It 
was  used  to  solicit  the  opinions  of  several  DERP  and  CERCLA 
experts  at  an  operational  decision  making  level  to  determine 
what  criteria  should  be  employed  in  the  DSS  and  the  relative 
iirqportance  to  be  assigned  to  each  of  those  criteria.  While 
the  Delphi  method  is  very  useful  for  gathering  the  decision 
criteria,  and  by  increasing  the  number  of  people  in  the 
procedure  the  range  of  ideas  generated  increases,  the 
analysis  can  become  unwieldy  (Saaty,  1982:234).  In  fact, 
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designing  a  methodology  for  analytically  determining  the 
group  utility  function  for  many  experts  often  cannot  be  done 
(Chan,  1993:65).  Therefore,  a  simple  additive  weighting 
(SAW)  multicriteria  decision  model  (MCDM)  was  pre-selected 
for  the  resulting  DSS,  and  the  experts  were  asked  to 
subjectively  determine  the  weights  to  be  assigned  to  each 
criterion.  As  Hwang  notes, 

...theory,  simulation  computations,  and  experience  all 
suggest  that  simple  additive  weighting  yields  extremely 
close  approximations  to  very  much  more  complicated 
nonlinear  forms,  while  remaining  far  easier  to  use  and 
understand.  (Hwang,  1981:103) 

More  will  be  said  about  the  implications  of  this  design  in 

Chapter  4 . 

The  objective  of  the  second  procedure  was  to  use  the 
strengths  of  multiattribute  utility  theory  (MAUT)  in  a 
questionnaire  format  to  determine  an  MCDM.  To  simplify  this 
procedure,  the  utility  function  of  one  decision  maker,  the 
head  of  the  Air  Force  Environmental  Restoration  Division, 
Colonel  James  Owendoff,  was  examined.  (It  should  be  noted 
that  Col.  Owendoff  worked  very  closely  with  his  DPM  expert, 
Mr.  Scott  Edwards,  in  responding  to  the  questionnaire. 
Additionally,  Mr.  Edwards  provided  responses  to  some  of  the 
follow-on  interview  questions.)  Unlike  the  Delphi 
technique,  pre-selected  decision  criteria  were  used  in  the 
questionnaire.  The  criteria  were  chosen  on  the  basis  of  the 
literature  review  and  a  pre-questionnaire  interview  with 
Col.  Owendoff.  While  this  method  does  not  inherently  build 
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stakeholder  support  for  the  findings  or  take  advantage  of 
the  power  of  group  decision  making  for  criterion  selection 
as  in  the  Delphi  technique,  it  also  doesn't  suffer  from 
having  to  assume  a  form  of  MCDM.  Instead,  the  results  of 
the  questionnaire  were  used  to  determine  the  form  of  model 
for  the  resulting  DSS. 

Delphi  Procedure 

This  section  outlines  the  methodology  to  be  used  in 
association  with  the  Delphi  method  for  eliciting  the 
opinions  of  experts.  Creating  a  DSS  using  this  approach 
requires  the  completion  of  two  phases.  The  first  phase  is 
the  determination  of  the  criteria  relevant  to  the 
prioritization  of  IRP  sites.  The  second  phase  is  the 
development  of  a  model  which  incorporates  the  preference 
structure  of  the  group,  and  which  will  bring  each  of  the 
criteria  into  a  common  scale,  called  utility. 

Phase., Qaa-z  BfiLermininq  Cr.i£exia.  The  determination  of 
discriminating  criteria  comprises  the  bulk  of  the  data 
collection  of  this  portion  of  the  research  and  consists  of 
several  steps : 

1)  Defining  the  population  of  interest, 

2)  Selecting  a  manageable-sized  sample  from  this 
population  which  represents  the  interests  of  the  entire 
population, 

3)  Developing  an  appropriate  data  gathering  instrument, 

4)  Testing  the  data  gathering  instrument,  and 
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5)  Collecting  data  using  the  data  gathering  instrument. 
These  steps  are  specifically  addressed  in  the  following 
sections . 

Population  of  Interest.  The  population  of 
interest  for  this  research  consists  of  all  stakeholders 
involved  in  the  restoration  of  contaminated  military  sites, 
including  the  Department  of  Defense  (DoD) ,  Congress,  the 
Environmental  Protection  Agency  (EPA) ,  individual  state 
regulators,  and  the  affected  public.  This  population 
determines  the  size  of  the  DoD  budget,  sets  priorities,  and 
ultimately  judges  DoD's  efforts  in  meeting  their  objectives. 

Sample  Selection.  Since  a  survey  of  the  entire 
population  is  not  feasible,  a  sample  of  the  population  must 
be  taken.  Initially,  a  judgment  (non-probability, 
purposive)  sample  composed  of  approximately  eight  to  ten  DoD 
Installation  Restoration  Program  (IRP)  experts  will  be 
selected.  Other  experts  in  IRP  and  CERCLA  management, 
including  experts  from  the  EPA  and  civilian  universities, 
will  be  identified  using  a  'snowball'  selection  method,  and 
will  be  encouraged  to  participate.  This  research  will 
include  the  opinions  of  other  stakeholders  because  these 
individuals  and  organizations  are  heavily  involved  in,  and 
have  significant  influence  over,  the  IRP  prioritization 
process . 

Due  to  this  relatively  small  sample  size,  each  selected 
individual  has  the  potential  of  greatly  affecting  the 
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results  of  the  data  collection  process,  and  thus  the 
confidence  in  the  resulting  MCDM.  Therefore,  respondents 
will  be  selected  on  the  basis  of  their  position  and 
experience  as  IRP  or  CERCLA  managers,  researchers,  or 
regulators.  While  a  judgment  sample  has  the  potential  of 
being  biased,  other  segments  of  the  population  generally  do 
not  have  the  background  knowledge  required  to  provide  the 
type  of  input  requested  in  this  process.  Instead,  the 
general  population  frequently  looks  to  regulatory  agencies 
to  decipher  the  technical  issues  associated  with  the  IRP. 
Additionally,  state  regulators  and  Congressional  members 
have  more  localized  interests  which  may  bias  the  survey  with 
factors  representative  of  the  sites  within  their 
jurisdiction.  For  these  reasons,  the  selected  body  of 
experts  will  be  relied  upon  to  represent  the  current 
paradigm  for  environmental  restoration  within  the  DoD. 

Development  of  the  Data  Gathering  Instrument.  For 
this  phase  of  the  research,  the  objective  of  the  data 
collection  process  is  to  gather  data  which  is  amenable  for 
use  as  input  data  in  a  simple  additive  weighting  (SAW)  MCDM. 
The  process  of  resolving  the  investigative  questions 
(presented  in  Chapter  One)  will  produce  the  necessary  data 
to  construct  this  model . 

The  Delphi  procedure  was  chosen  as  the  best  means  to 
answer  the  investigative  questions.  The  Delphi  is  a 
heuristic  for  the  gathering  of  group  opinions  and  is 
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particularly  well  suited  for  eliciting  the  opinions  of 
experts  (Brown,  1968:3).  The  iterative  nature  of  the  Delphi 
technique  has  several  advantages  for  data  collection  over 
other  standard  questionnaire  surveys.  The  Delphi  technique 
takes  advantage  of  the  benefits  of  group  thinking,  such  as 
brainstorming  and  idea  construction,  as  well  as  providing 
capacity  to  clarify  both  questions  and  responses  in  order  to 
reach  a  consensus  among  participants.  Simultaneously,  by 
avoiding  face-to-face  discussion,  the  technique  eliminates 
the  drawbacks  of  group  thinking,  such  as  domination  by  a 
particular  individual,  extraneous  noise,  reluctance  to 
change  a  publicly  stated  opinion,  compromise,  and  the  desire 
to  conform  to  majority  opinion.  (Brown,  1968:2;  Dalkey, 
1967:3;  Dalkey,  1969:14) 

The  Delphi  technique  consists  of  three  basic  features 
which  are  intended  to  increase  validity  and  reliability. 

In  general,  Delphi  procedures  have  three  features: 

(1)  anonymity,  (2)  controlled  feedback,  and  (3)  statistical 
group  response.  Anonymity,  effected  by  the  use  of 
questionnaires  or  other  formal  communication  channels,  such 
as  on-line  computer  communication,  is  a  way  of  reducing  the 
effect  of  dominant  individuals.  Controlled  feedback  - 
conducting  the  exercise  in  a  sequence  of  rounds  between 
which  a  summary  of  the  results  of  the  previous  round  are 
communicated  to  the  participants  -  is  a  device  for  reducing 
noise.  Use  of  statistical  definition  of  the  response  is  a 
way  of  reducing  group  pressure  for  conformity ... (Dalkey, 
1969:16) 

Use  of  the  Delphi  technique  with  groups  of  experts  has 
been  shown  to  provide  reliable  results  to  the  extent  that 
results  tend  to  be  reproducible  (Dalkey,  1969:12).  This  is 
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especially  true  when  the  goal  of  the  process  is  to  determine 
a  numerical  estimate  of  the  median  value  of  individual 
estimates,  even  though  individual  estimates  may  exhibit 
considerable  variance  (Dalkey,  1967:3;  Dalkey,  1969:10). 

Testing  the  Data  Gathering  Instrument.  While  a 
Delphi  questionnaire  primarily  consists  of  open-ended 
questions,  some  criteria  discovered  in  the  background 
research  will  be  included  in  the  questionnaire  to  help 
clarify  its  intent.  This  should  also  reduce  the  number  of 
iterations  required  to  identify  and  give  weight  to  all  the 
discriminating  criteria  and  allow  more  time  for  reaching  a 
consensus.  Any  biases  introduced  by  providing  potential 
responses  will  be  minimized  by  allowing  respondents  to 
strike  out  any  suggested  criteria  they  feel  are 
inappropriate.  Prior  to  sending  the  questionnaires,  each 
iteration  of  the  Delphi  will  be  presented  to  members  of  the 
faculty  to  ensure  that  the  intent  of  each  question  is 
understandable  and  elicits  the  appropriate  type  of  response. 

Collection  of  Data.  The  Delphi  technique  will  be 
performed  by  sending  questionnaires  to  the  selected  experts, 
gathering  and  summarizing  responses,  providing  feedback  to 
participants,  and  performing  subsequent  iterations 
consisting  of  either  refined  questions  or  new  questions 
which  may  evolve  as  a  result  of  observations  from  previous 
iterations,  or  a  combination  of  both.  A  maximum  of  three 
iterations  will  be  used  to  accomplish  the  objectives  of  this 
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portion  of  the  research.  If  further  information  is 
required,  in  the  interests  of  time,  a  face-to-face  group 
discussion  may  be  held  at  The  Air  Force  Institute  of 
Technology  (AFIT)  to  finalize  the  results. 

In  performing  the  multiple  iterations  of  the  Delphi 
technique,  some  data  analysis  is  inherently  performed.  This 
occurs  as  a  result  of  compiling  responses  and  revising 
questions  for  the  subsequent  rouno  of  questioning. 

Therefore,  upon  completion  of  the  Delphi,  very  little 
manipulation  of  the  data  will  be  required  to  construct  the 
simple  additive  weighting  decision  support  model. 

Phase  Two  -  Model  Development .  This  phase  of  the 
research  is  concerned  with  the  development  of  a  decision 
support  model  commensurate  with  the  level  of  data  elicited 
from  the  experts.  This  will  be  accomplished  by  first 
determining  the  preference  structure  of  the  experts  involved 
in  Phase  One  which  will  indicate  a  utility  function  for  each 
criterion.  Combining  the  utility  functions  of  the  criteria 
with  the  weighting  factors  also  determined  in  Phase  One  will 
produce  the  decision  support  model.  These  topics  are 
described  more  fully  in  the  following  sections.  Also,  a 
discussion  of  how  this  model  will  be  evaluated  will  be 
presented  following  the  MAUT  methodology. 

Determination  of  Preference  Structure  and  Utility 
Functions .  Preference  structure  refers  to  the  set  of  rules 
which  govern  the  processing  of  discriminating  criteria 
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within  a  model.  The  preference  structure  brings  each  of  the 
criteria  into  a  common  scale  called  utility.  For  this 
portion  of  the  research,  the  determination  of  the  group 
preference  structure  will  consist  of  sending  out  a  round  of 
questions  in  a  final  iteration  of  the  Delphi  technique. 

This  iteration  will  have  the  experts  specifically  identify 
how  they  value  each  criterion  within  the  range  of  possible 
values.  The  responses  to  these  questions  will  be  used  to 
arrive  at  a  median  utility  function  for  each  criterion.  The 
utility  function  can  be  depicted  as  a  graph  showing  how  the 
experts*  utility  decreases  as  conditions  favoring 
prioritization  decrease. 

Development  of  a  Decision  Support  Model.  As 
mentioned  previously,  this  portion  of  the  research  assumed 
preferential  independence  of  the  criteria  Therefore,  a 
simple  additive  weighting  model  is  applicable  as  a  means  of 
analyzing  and  coirparing  the  criteria  and  prioritizing  the 
projects  (Chan,  1993:33).  The  IRP  projects  will  be  scored 
with  a  value  function  that  is  in  the  form: 

Project  Score  =  £  v^v.tyj  (1) 

Where  w4  represents  the  weighting  factor  for  criterion  i, 
and  v4(yj  represents  the  criterion-specific  utility  for  the 
value  yt  (Chan,  1993:33).  The  utility  term  can  be 
simplified  to  u4  yielding  the  form: 
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(2) 


Project  Score  =  ]£  w,u; 

The  weighting  factor  will  be  transformed  to  a  scale  of 
0  to  1,  where  the  sum  of  the  weighting  factors  equals  1. 

The  utility  factors  will  remain  essentially  unchanged  from 
the  form  which  is  returned  by  the  Delphi  participants,  and 
will  be  on  a  scale  of  0  to  100.  Therefore,  the  final  score 
for  any  project  to  be  evaluated  will  be  on  the  scale  of  0  to 
100.  This  should  facilitate  ease  of  use  and  understanding 
for  this  model  because  it  brings  the  values  of  several 
criteria  into  a  common  scale,  it  relates  the  project  score 
on  a  scale  which  is  familiar,  and  it  promotes  the  ability  to 
distinguish  trends  in  the  data. 

While  this  method  of  obtaining  weights  is 
straightforward  and  simplifies  the  computations,  it  forces 
the  experts  to  simultaneously  analyze  the  relationships  of 
many  variables  This  weakness  is  analogous  to  rank  ordering 
a  group  of  items  from  heaviest  to  lightest  without  a  scale. 
It  is  very  difficult  io  correctly  order  them  by  picking  up 
each  item  and  assigning  them  a  weic  .c.  It  is  much  easier  to 
pick  up  two  items  and  do  a  pairwise  comparison,  continuing 
this  process  until  all  pairs  have  been  assessed.  The  MAUT 
procedure  overcomes  this  weakness.  (Saaty,  1980:6,  33) 
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MftUT  £ rocedure 

MAUT  has  recently  found  favor  among  researchers  for  its 
ability  to  enhance  decision  making.  As  Saaty  notes,  the 
process  helps  decision  makers  organize  their  thought 
patterns  and  reach  conclusions.  It  allows  us  to  simplify 
and  structure  a  system  into  its  interacting  components,  and 
to  synthesize  the  components  by  measuring  and  ranking  their 
impact  on  the  decision  at  hand.  Furthermore,  MAUT  is 
capable  of  converting  qualitative  data  into  a  quantitative 
measures  that  permit  comparisons  and  evaluations.  (Saaty, 
1982:2-6,  202) 

There  are  four  basic  steps  to  any  decision  analysis 
problem.  The  first  step  involves  structuring  the  problem, 
or  the  identification  of  goals  and  criteria.  Next  is  the 
model  formulation  step.  The  third  step  is  eliciting  the 
relationships  and  values  of  the  criteria  from  the  decision 
makers  or  experts,  and  last  is  the  evaluation  of  the 
resulting  model.  (Barron  and  Kleinmuntz,  1986:291) 

Problem  Structure.  The  first  task  in  MAUT  policy 
analysis  is  to  identify  the  goals/objectives  and  criteria  to 
use.  Morgan  identifies  three  distinct  types  of  objectives 
for  use  in  risk  management  policy  analysis.  They  are: 

1)  Utility-based  objectives  which  all  involve  decisions 
based  on  the  valuation  of  outcomes .  This  category  includes 
cost-benefit,  cost-effectiveness,  maximum  utility,  and  the 
like . 

2)  Rights -based  objectives  which  are  less  concerned 
with  the  outcome  as  they  are  with  the  process  or  allowed 
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activities.  Zero  risk,  compensation  for  risk,  and 
constrained  risk  are  examples  of  rights-based  objectives. 

3)  Technology-based  objectives  which  are  frequently 
used  in  environmental  regulations.  An  example  of  this 
objective  is  reducing  risk  to  the  extent  the  best  available 
technology  allows.  {Morgan,  1990:25-28) 

A  utility-based  objective  best  fits  this  research  problem. 
Briefly  restated,  the  objective  is  to  maximize  the  utility 
associated  with  important  project  attributes  given  the 
fiscal  constraints  imposed  by  the  Defense  Environmental 
Restoration  Account  (DERA)  budget. 

The  decision  one  makes  as  to  what  criteria  to  include 
in  the  problem  and  which  to  leave  out  is  among  the  most 
important  and  difficult  decisions  to  be  made  in  quantitative 
policy  analysis  (Morgan  and  Henrion,  1990:31).  As  the 
literature  review  demonstrated,  there  are  a  host  of  relevant 
criteria  and  little  agreement  on  which  to  employ  in  a  model . 
There  are,  however,  several  good  reasons  for  limiting  the 
number  of  criteria  in  the  model.  First  of  all,  for  every 
criterion  used,  a  metric  has  to  be  developed  which  will 
apply  to  the  wide  spectrum  of  projects  that  make  up  the  IRP 
(studies,  management,  design,  construction,  monitoring, 
research  and  development,  etc.)  The  choice  of  the  metric 
itself  can  be  controversial  and  may  detract  from  the  model's 
credibility  or  the  utility  associated  with  a  given 
criterion.  Secondly,  the  difficulty  of  maintaining 
independence  among  the  criteria  is  proportionate  to  the 
number  of  criteria.  The  lack  of  independence  leads  to  a 
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highly  complex  modeling  task  (Chan,  1993:40).  The  decision 
maker  will  have  difficulty  assessing  his  or  her  utility  for 
a  variable  in  the  presence  of  many  possible  compensatory  or 
noncompensatory  variables,  and  the  analyst  will  have  more 
difficulty  designing  the  questionnaire  and  computing  the 
results  to  elicit  the  factors  and  weights  associated  with 
that  utility.  Finally,  the  documentation  that  will  have  to 
be  submitted  with  each  project  and  analyzed  by  the  decision 
maker  will  also  increase  proportionately  with  the  number  of 
criteria  chosen.  Since  the  Air  Force  has  over  4,000  sites, 
this  could  be  a  significant  factor  in  the  usefulness  of  the 
resulting  DSS. 

With  these  difficulties  in  mind,  it  was  decided  that 
the  possible  criteria  identified  earlier  should  be  collapsed 
into  three  or  four  criteria  that  would  be  used  in  the  final 
MCDM.  The  final  criteria  should  be  widely  mentioned  in  the 
literature.  Additionally,  similar  criteria  should  be 
combined  so  as  to  maintain  independence  among  the  criteria. 
Finally,  a  semi-structured  interview  with  Col.  Owendoff  was 
employed  to  elicit  the  decision  maker's  preferences  for 
variables  and  metrics  before  the  design  of  the 
questionnaire.  A  copy  of  the  interview  guide  and  responses 
can  be  found  at  Appendix  C.  The  description  of  the  final 
criteria  selected  and  the  metrics  chosen  to  represent  them 
follow. 
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Site  risk.  This  criterion  was  identified  by  Col. 
Owendoff  as  an  important  consideration,  was  mentioned  in 
most  of  the  previous  studies,  is  a  statutory  consideration, 
and  is  also  emphasized  in  Air  Force  regulations  and 
guidance.  A  project  associated  with  a  high  risk  site  is 
assumed  to  be  strongly  correlated  with  protection  of  human 
health  and  the  environment,  risk  reduction,  and  reduced 
future  liability. 

Site  risk  will  be  measured  using  a  DPM  score.  The  DPM 
was  developed  to  assist  DoD  in  prioritizing  sites  and  is  a 
reflection  of  the  overall  risk  at  the  site.  While  a 


reduction  in  risk,  as  examined  by  Edwards  in  1992,  is 
perhaps  a  better  metric  for  projects  like  removal  actions 
and  remedial  actions,  other  categories  of  projects  have  no 
potential  to  reduce  risk.  Therefore,  the  site  risk 
represented  by  the  DPM  score  will  be  used  for  all  projects. 

Malar  -Command  (MAJCQM)  priority.  This  measure  is 
an  attempt  to  combine  several  subjective  criteria  mentioned 
in  the  literature  while  recognizing  the  difficulty  of 
establishing  a  meaningful,  quantitative  measure  to  those 
criteria.  Furthermore,  Col.  Owendoff  indicated  that  the 
MAJCOMs  are  frequently  the  true  decision  makers  in  that  they 
must  allocate  the  funds  provided  by  higher  headquarters  to 
the  projects  under  their  jurisdiction  they  feel  are  most 
deserving.  He  also  noted  that  the  MAJCOMs  are  closer  to  the 
operating  level  and  have  a  better  grasp  of  the  site  demands. 
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To  implement  this  criterion,  MAJCOMs  with  50  or  more 
sites  would  be  asked  to  rank  the  projects  submitted  each 
fiscal  year  on  the  basis  of  the  urgency  of  the  requirement 
(whether  or  not  it  is  required  in  the  current  fiscal  year  by 
statute,  agreement,  or  as  a  result  of  other  pending  actions 
like  new  construction  on  the  site) ,  the  feasibility  of  the 
technology  to  be  employed,  the  political  sensitivity  of  the 
site,  compliance  with  established  planning  documents  and 
concurrence  of  the  technical  review  committee  and  affected 
public,  environmental  equity  (site  goals  are  commensurate 
with  other  sites) ,  and  overall  compliance  with  Air  Force 
goals  (10%  of  the  sites  closed  each  year,  60%  of  the  funds 
spent  on  remedial  actions,  and  complete  all  sites  by  the 
year  2000)  . 

The  MAJCOM  priority  will  be  converted  into  a  value 
ranging  from  zero  to  one  with  the  following  equation: 

Value  =  1  -  [priority  /  total  number  of  projects]  (3) 

A  MAJCOM' s  highest  priority  project  will  have  a  value  almost 
equal  to  one.  The  lowest  priority  project  will  have  a  value 
of  zero.  Smaller  MAJCOMs  and  other  activities  with  fewer 
than  50  sites  could  have  their  projects  evaluated  and 
prioritized  by  a  committee  from  the  Air  Force  Center  for 
Environmental  Excellence  (AFCEE)  to  make  the  overall  ratios 
commensurate  with  the  larger  MAJCOMs.  AFCEE  has  a  good 
understanding  of  overall  Air  Force  requirements  and  provides 
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technical  support  to  many  of  the  smaller  commands.  Since 
this  process  establishes  a  zero  to  one  scale  and  establishes 
mid-management’s  utility  for  the  projects,  the  questionnaire 
will  only  be  used  to  determine  the  decision  maker’s 
preference  for  the  weight  to  given  to  this  criterion  in  the 
overall  utility  function. 

Cost .  The  need  for  a  prioritization  model  is 
predicated  on  an  overall  funding  shortfall.  The  literature 
consistently  cited  cost  effectiveness  as  an  important  factor 
to  the  resulting  decision,  and  several  studies  used  cost- 
benefit  ratios  to  prioritize  the  alternatives.  This  model 
will  not  seek  to  optimize  the  ranking  of  projects  by  using 
costs  in  this  manner  since  cost-benefit  ratios  implicitly 
assume  linear  utility  functions  with  ratio-level  data.  One 
possible  problem  with  this  assumption  concerns  the  magnitude 
of  the  potential  disparity  between  the  highest  cost  project 
and  the  lowest.  This  difference  could  reduce  the 
discriminatory  power  for  low  and  moderate  cost  projects 
(e.g.,  if  the  most  expensive  project  cost  $10  million  and 
the  scale  was  based  on  total  cost  divided  by  the  most 
expensive  project,  all  projects  under  $1  million  would  be 
scaled  between  0.9  and  1.0).  Another  problem  would  be 
created  if  the  measurement  of  benefits  were  not  represented 
on  an  interval  scale.  Without  an  interval  scale,  division 
by  a  scalar  such  as  cost  changes  the  ratio  of  two 
measurements  (Chankong,  1983:30)  and  renders  utility 
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functions  meaningless.  Furthermore,  cost-benefit  ratios 
overlook  other  possible  covariance  relationships  among  the 
criteria.  Therefore,  a  cost-benefit  ratio  will  not  be 
assumed.  Instead,  the  relationship  to  other  criteria  in  the 
overall  value  function,  its  relative  weight  in  that 
relationship,  and  the  univariate  utility  function  for 
project  costs  will  be  determined  from  the  decision  maker's 
responses  to  the  questionnaire. 

Model  Formulation.  There  are  many  types  of  decision 
models  which  can  be  used  to  solve  problems  composed  of 
multiple  criteria  or  attributes.  The  class  of  model  chosen 
depends  mainly  upon  the  type  and  structure  of  data  available 
for  use,  as  shown  in  Figure  2.  The  choices  of  models  may 
generally  be  separated  into  the  two  categories  of 
noncompensatory  models  and  compensatory  models. 

Noncompensatory  models  are  MCDMs  which  do  not  allow  for 
trade-offs  between  criteria.  This  means  that  a  deficient 
value  in  one  attribute  cannot  be  compensated  by  an  abundance 
in  another  attribute.  Scores  are  calculated  on  an 
attribute-by-attribute  basis.  Models  in  this  category  are 
dominance,  maximin,  maximax,  conjunctive  constraint  method, 
disjunctive  constraint  method,  and  lexicographic  ordering. 
(Schooff,  1990:22)  The  questionnaire  results  will  determine 
if  a  noncompensatory  model  is  appropriate;  however,  given 
the  nature  of  the  criteria,  this  result  is  unlikely. 
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Figure  2.  A  Taxonomy  of  Multi-Attribute  Decision 


Making  Methods  (Hwang,  1981:6). 


Compensatory  models  are  MCDMs  which  permit  trade-offs 
between  criteria,  or  stated  differently,  changes  in  one 
attribute  can  be  offset  by  changes  in  other  attributes 
(Hwang,  1981:25) .  This  type  of  decision  model  requires  that 
a  preference  structure  be  determined  by  the  decision  maker. 
Typically,  this  is  done  by  assigning  a  set  of  weights  to 
criteria  through  pairwise  comparisons. 

Compensatory  models  are  further  divided  into  three  sub¬ 
categories  : 

1)  Scoring  models.  These  models  select  alternatives  on 
the  basis  of  highest  score.  This  requires  a  well  defined 
preference  structure  which  assigns  values  to  each  criterion 
and  tallies  them  for  a  final  score.  Models  in  this  sub- 
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category  include  simple  additive  weighting  (SAW),  analytical 
hierarchy  process  (AHP) ,  and  interactive  simple  additive 
weighting . 

2)  Compromising  models.  These  models  select 
alternatives  based  upon  their  distance  from  some  ideal 
score.  This  approach  is  referred  to  as  goal  setting  (Chan, 
1991:13).  This  sub-category  includes  the  Technique  for 
Order  Preference  by  Similarity  to  Ideal  Solution  (TOPSIS) 
model . 

3)  Concordance  models.  These  models  arrange  a  set  of 
preferences  which  best  satisfy  a  given  concordance  measure 
(Schooff,  1990:23).  Models  in  this  sub-category  include 
permutation  method  and  linear  assignment  method. 

The  outcome  of  the  questionnaire  will  determine  the 
exact  model  used;  however,  based  on  the  objectives  of  the 
research  and  the  criteria  chosen  for  inclusion  in  the  model, 
it  is  anticipated  that  the  class  of  MCDM  selected  will  be  in 
the  compensatory  category,  and  further,  that  the  model  will 
be  a  scoring  model  such  as  SAW  or  AHP. 

Eliciting  Relationships  an<L_Values .  Once  the  criteria 
of  interest  have  been  determined,  the  research  task  focuses 
on  the  value-function  measurement  task.  This  five  step 
process  has  been  described  by  Zelany  as  follows: 

1)  Familiarize  the  decision  maker  with  the  concepts  and 
techniques  of  the  process . 

2)  Identify  the  form  of  the  utility  function.  This 
step  involves  determining  preferential  independence  (the 
trade-off  between  two  variables  is  not  affected  by  a  third 
variable)  and  mutual  utility  independence  (the  relative 
utility  of  a  variable  remains  the  same  regardless  of  other 
variables)  of  the  criteria.  An  additive  value  function 
requires  preferential  independence,  whereas  both  types  of 
independence  are  required  for  multiplicative  models. 

3)  Measure  the  component  value  functions.  This  step 
requires  utility  calculations  on  each  criteria  independent 
of  the  other  criteria. 
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4)  Determine  the  scaling  factors  and  relative  weights 
to  be  applied  to  the  criteria.  The  scaling  factors  allows 
all  the  criteria  to  be  represented  on  a  scale  of  zero  to  one 
so  that  one  criterion  does  not  dominate  another  simply  on 
the  basis  of  the  representative  metric.  The  relative 
weights  are  indicative  of  the  importance  of  each  criterion 
on  the  overall  score. 

5)  Validate  the  model  against  observed  rankings.  This 
step  will  be  described  in  the  next  section  as  part  of  the 
evaluation  of  results.  (Chan,  1993:32,40-44) 

The  pre-questionnaire  administered  to  Col.  Owendoff 
accomplished  step  one  of  this  five-step  process.  The  data 
for  the  remaining  analysis  was  collected  via  the  research 
questionnaire  (shown  with  responses)  found  at  Appendix  D. 
Section  A,  questions  1-6  of  the  questionnaire  were  designed 
to  test  for  both  preferential  independence  and  the  weights 
associated  with  each  of  the  criteria.  Section  B,  questions 
7-9  tested  for  mutual  utility  independence.  The  next  three 
questions.  Section  C,  were  designed  to  elicit  the  certainty 
equivalents  for  several  ranges  of  DPM  scores  and  costs  so 
that  a  utility  function  could  be  determined  for  each. 

Section  D  provided  a  check  on  the  responses  to  Section  A  by 
gathering  additional  data  required  for  determination  of  the 
consistency  of  the  weights  associated  with  each  of  the  three 
criteria.  Finally,  Section  E  was  added  as  a  check  on  the 
questionnaire.  The  results  of  the  analysis  can  be  compared 
to  the  answers  provided  for  question  15  to  determine  if  they 
make  sense.  Question  16  is  a  check  on  the  validity  of  the 
criteria  used  for  the  model,  and  question  17  was  an  attempt 
to  verify  the  assumption  that  the  projects  being  ranked  by 
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the  model  should  be  treated  as  discrete  units  rather  than 
continuous  variables.  If  the  decision  maker  believes 
project  managers  have  great  flexibility  in  scaling  back 
their  projects,  then  the  assumption  that  the  projects  should 
be  treated  as  discrete  units  (all  or  none)  may  not  hold. 
Continuous  variables  can  be  subjected  to  much  more  rigorous 
analytical  methods  involving  marginal  analysis, 
optimization,  and  the  like. 

EvaluaLiftiL  ..of.  P.aJ,ph.i  and  MftUT  -.Results 

As  Morgan  and  Henrion  appropriately  noted,  policy 
analysis  models  often  are  difficult  or  inpossible  to 
empirically  validate  (1990:18).  The  prioritization  problem 
which  this  research  is  trying  to  overcome  is  a  recent 
phenomenon  and  existing  models  are  not  available  for 
comparison  of  results.  Also,  because  current  decisions  are 
made  on  the  basis  of  a  'fair  share  allocation',  a 
prioritized  listing  of  actual  projects  is  not  available. 
Furthermore,  many  of  the  criteria  used  in  the  DSS  are  not 
submitted  with  the  project  documentation  currently  needed 
for  validation  and  approval. 

These  problems  were  not  unexpected,  and  part  of  the 
justification  for  two  research  methodologies  lay  in 
providing  a  means  of  comparison  of  results.  As  a  method  of 
evaluation,  the  output  of  the  two  models  will  be  conpared. 
For  the  purposes  of  this  research,  a  set  of  actual  IRP 
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projects  will  be  used  to  compare  model  rankings.  Where  data 
gaps  exist,  information  will  be  assumed  from  the  information 
available,  and  possibly  gathered  from  IRP  managers  familiar 
with  the  individual  projects.  A  comparison  of  the  resultant 
rankings  will  give  an  indication  of  the  validity  of  the  two 
prioritization  methods. 

Summary 

This  chapter  has  presented  the  proposed  methodologies 
for  this  research.  The  processes  of  determination  of  the 
discriminating  criteria  and  development  of  appropriate  MCDMs 
are  anticipated  to  result  in  a  much  needed  decision  support 
system.  Data  collection  results  and  analyses  performed 
under  this  research,  as  well  as  concluding  statements  will 
be  presented  in  the  subsequent  chapters. 


The  results  of  the  methodology  employed  in  Chapter  3 
are  documented  in  this  chapter  of  the  research.  The  Delphi 
results  are  presented  separately  from  the  multiattribute 
utility  theory  (MAUT)  results.  This  chapter  is  concluded 
with  a  comparison  of  the  two  procedures. 

Analysis  of  Data  from  the  Delphi  Procedure 

The  research  performed  in  association  with  the  Delphi 
procedure  consisted  of  three  rounds  of  questionnaires.  Each 
round  was  sent  to  a  group  of  experts  with  experience  in 
prioritizing  DERP  projects  or  CERCLA  management.  The  first 
two  rounds  established  the  criteria  the  experts  felt  to  be 
pertinent  to  the  prioritization  of  DERP  projects  and  the 
weights  associated  with  those  criteria.  The  third  round  of 
the  Delphi  established  a  preference  structure  for  values 
within  each  of  the  criteria.  Using  these  criteria, 
weighting  factors,  and  preference  structures,  a  decision 
support  model  will  be  used  to  assign  scores  to  DERP  projects 
for  comparison  and  prioritization.  The  results  and  analysis 
of  data  from  each  round  of  the  Delphi  procedure,  the 
subsequent  construction  of  the  decision  support  model,  and 
the  results  of  testing  the  model  will  be  discussed  more 
fully  in  the  following  sections. 


69 


Delphi  Round  One.  This  portion  of  the  Delphi  procedure 
consisted  of  identifying  a  group  of  experts  to  be  used  as  a 
sample  of  the  population  of  interest  and  familiarizing  these 
experts  with  the  goals  of  the  research,  collecting  first- 
round  data  identifying  the  criteria  which  these  experts  felt 
were  important  in  prioritizing  IRP  projects,  and  summarizing 
the  individual  data  collected  from  the  first  questionnaire 
for  use  in  the  second  round. 

Identifying  Experts  for  Use  in  the  Delphi 
Analysis .  As  indicated  previously,  the  population  from 
which  a  sample  will  be  selected  for  this  research  consists 
of  DERP  and  CERCLA  experts  familiar  with  the  problems  of 
prioritizing  DERP  projects.  A  list  of  Air  Force  MAJCOM- 
level  DERP  experts,  and  others  with  knowledge  in  the  DERP 
process,  was  provided  by  Captain  Rob  Wilson,  an  instructor 
at  AFIT/DE,  who  has  had  the  opportunity  to  work  with  these 
individuals  for  the  purposes  of  education  and  dissemination 
of  information  on  recent  DERP  issues.  Prior  to  sending  out 
the  first  questionnaire,  each  expert  was  contacted  and  given 
a  general  background  of  the  research,  and  the  objectives  of 
the  Delphi  procedure.  Attempts  were  made  to  contact  each  of 
the  MAJCOM-level  experts  identified  by  Captain  Wilson. 

From  the  list,  eight  experts  agreed  to  participate  in 
the  study.  The  list  of  names  provided  by  Captain  Wilson  is 
included  at  Appendix  E.  Two  additional  respondents  from 
non-Air  Force  institutions  were  also  identified  and  included 
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in  this  round  of  the  Delphi  procedure.  These  individuals 
have  expertise  in  other  aspects  of  DERP  and  CERCLA,  and  do 
not  necessarily  have  the  same  priorities  as  the  Air  Force  or 
DoD.  They  were  included  in  an  attempt  to  incorporate  a 
wider  perspective  on  the  DERP  prioritization  process  and  to 
represent  the  many  non-Air  Force  and  non-DoD  factions  which 
have  influence  in  this  process. 

Collecting  First  Round  Data.  In  order  to  minimize 
the  number  of  iterations  required  to  elicit  usable  data  from 
the  experts,  and  to  further  clarify  the  intent  of  the 
questionnaire,  the  first  round  of  the  Delphi  included  nine 
potential  criteria  taken  from  the  Nationa1  Contingency  Plan 
(NCP) ,  40  CFR  300.430(e)(9).  These  criteria  are  meant  to  be 
used  to  compare  remediation  alternatives  and  to  evaluate  the 
appropriateness  of  response  actions  in  the  IRP  process. 
However,  the  criteria  are  also  applicable  to  other  stages  of 
the  clean-up  process  (e.g.,  studies  and  investigations). 

The  experts  were  asked  to  examine  these  criteria  for 
their  appropriateness  in  prioritizing  DERP  projects,  with 
the  option  of  adding  any  criteria  they  felt  were  important, 
or  deleting  any  criteria  they  felt  were  not  important.  The 
experts  were  also  asked  to  give  a  brief  justification  for 
each  addition  or  deletion,  and  give  a  means  of  measurement 
for  each  criterion  included.  Finally,  the  experts  were 
asked  to  assign  a  weight  to  each  criterion  which  reflected 
its  importance  relative  to  each  of  the  other  criteria. 
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The  cover  letter,  distribution  list,  and  questionnaire 
from  the  first  round  of  the  Delphi  are  included  at  Appendix 
F. 

Summarizing  the  Data  from  Round  One.  Of  the  ten 
experts  included  in  the  first  round,  eight  responses  were 
gathered.  Table  1  shows  the  criteria  which  were  identified 
by  the  experts  and  the  weight  assigned  to  each  criterion. 

In  summarizing  this  round  of  results,  the  mean  was  used  to 
rank  the  relative  importance  of  each  criterion.  In  the 
subsequent  rounds  of  the  Delphi,  the  median  response  was 
used  to  arrive  at  the  final  weight. 

Table  1 

Round  One  Summary:  Criteria  and  Weights 


Expert  Number _ 

Average 


Criteria: 

1 

2 

3 

4 

5 

6 

7 

S 

Weight 

1 

Protection  of  HH  ft  E 

90 

100 

97 

100 

100 

too 

73 

2 

Implemcotabdaty 

95 

(5 

95 

70 

70 

49 

71 

67 

3 

Coat 

40 

95 

17 

50 

60 

50 

60 

SO 

65 

4 

Long  Term  Eflecrivrneea 

50 

90 

94 

40 

SO 

65 

89 

64 

5 

Community  Acceptance 

90 

98 

65 

90 

20 

25 

73 

58 

6 

Reduction  in  T,  M  ft  V 

95 

65 

75 

70 

70 

70 

56 

7 

Short  Term  Efiecriveneu 

100 

50 

70 

30 

70 

50 

50 

53 

8 

State  Acceptance 

65 

90 

95 

30 

30 

75 

48 

9 

Compliance  with  ARARi 

70 

SO 

40 

SO 

20 

90 

48 

10 

RuktoHHAE 

100 

100 

25 

II 

Future  Land  U*e 

99 

50 

19 

12 

Ruk  Axecumert 

IOO 

13 

13 

Federal  Facility  Agreement 

IS 

10 

12 

14 

Protection  of  Reeourcea 

SO 

10 

13 

Pollution  Prevention 

70 

9 

16 

Inatitutional  Control  of  Rak 

50 

6 

17 

EPA  Acceptance 

35 

4 

It 

Completion  of  Work,  in  Prog. 

25 

3 

19 

Political  Facton 

20 

3 

20 

Socioeconomic  Impacts 

10 

1 

21 

Minion  Impact 

10 

1 
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Delphi  Round  Two.  This  portion  of  the  Delphi  procedure 
consisted  of  developing  and  sending  out  a  second 
questionnaire  which  served  two  purposes:  1)  reporting  the 
summarized  results  of  the  first  round  of  the  Delphi,  and  2) 
collecting  refined  second-round  data  for  the  criteria 
important  in  prioritizing  IRP  projects.  This  round  was 
concluded  by  summarizing  the  new  data  collected  from  the 
second  questionnaire. 

Developing  the  Second  Questionnaire.  In  order  to 
focus  on  the  issues  involved  in  the  prioritization  process 
and  bring  the  experts  toward  consensus  on  a  limited  number 
of  criteria,  the  data  gathered  in  the  first  round  were 
collapsed  into  eight  new  criteria.  These  new  criteria  were 
deduced  by  the  researchers  through  inspection  of  the 
individual  criteria  returned  in  the  first  round  of  the 
Delphi.  The  basis  for  grouping  several  criteria  into  a 
single  criterion  was  taken  mainly  from  similarities  in  the 
experts'  recommended  method  of  measurement  for  the  criteria. 

Collecting  Second  Round  Data.  The  second  Delphi 
questionnaire  permitted  the  experts  to  examine  the 
collective  results  of  the  entire  group,  to  examine  the 
restructured  criteria,  and  to  re-weight  the  new  criteria  in 
light  of  this  feedback.  Included  in  the  second 
questionnaire  was  a  brief  definition  of  each  criterion  and  a 
proposed  method  of  measurement  based  upon  the  suggestions 
from  the  first  round.  The  experts  were  asked  to  examine 
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these  definitions  and  give  any  comments  they  may  have 
regarding  potential  problems.  Prior  to  this  round,  two 
additional  non-Air  Force  experts  were  added  to  the  original 
ten  experts  participating  in  the  Delphi  procedure. 

The  cover  letter,  distribution  list,  and  questionnaire 
from  the  second  round  of  the  Delphi  are  included  at  Appendix 
G. 

Summarizing  the. -Data  from  Round  Two.  Responses  to 
the  second  Delphi  questionnaire  were  received  from  seven  of 
the  twelve  experts.  Table  2  shows  how  the  experts  assigned 
weights  to  each  of  the  new  criteria.  Based  upon  the  fact 
that  no  other  criteria  were  introduced  (or  re-introduced) , 
no  additional  iterations  were  performed  to  attempt  to 
further  refine  the  criteria  or  their  associated  weights. 

Table  2 

Round  Two  Summary:  Criteria  and  Weights 


Expert  Number 


Criteria: 

t 

2 

3 

4 

9 

6 

7 

1 

Threat  to  HH  &  E 

too 

100 

100 

99 

100 

2 

Expected  Reduction  of  Risk 

mM 

H 

99 

90 

99 

80 

3 

TechnicaJ  Feasibility 

u 

70 

90 

99 

70 

98 

70 

4 

Cost 

99 

80 

97 

90 

9 

Federal  Facility  Agreement 

30 

89 

10 

90 

70 

93 

90 

6 

Community  Acceptance 

SO 

69 

90 

89 

30 

96 

40 

6 

Mission  Impact 

0 

80 

79 

90 

100 

20 

8 

Socioeconomic  Impacts 

10 

0 

10 

80 

30 

94 

20 

74 


Delphi  Round  Three.  This  portion  of  the  Delphi 
procedure  consisted  of  developing  and  sending  out  the  third 
Delphi  questionnaire  to  collect  data  pertinent  to  the 
experts'  preference  structure  for  data  points  within  each  of 
the  criteria  identified  during  the  previous  rounds  of  the 
Delphi.  All  analysis  of  the  results  of  the  third 
questionnaire,  and  further  analysis  of  the  data  received  in 
the  second  questionnaire  will  be  discussed  later  in  the 
section  on  development  of  the  decision  support  model. 

Developing  the  Third  Questionnaire.  Having 
gathered  data  for  the  criteria  and  weights  associated  with 
the  project  prioritization  problem,  a  preference  structure 
was  needed  in  order  to  bring  various  data  points  within  each 
criterion  into  a  common  scale.  The  Round  Three 
questionnaire  was  designed  for  this  purpose.  The  third 
questionnaire  also  reported  the  compiled  results  of  the 
previous  round,  however,  no  further  changes  were  requested 
for  the  criteria  or  weights. 

In  the  third  questionnaire,  the  experts  were  given  a 
number  of  data  points  within  each  criterion,  and  were  asked 
to  assign  their  utility  for  each  data  point.  The  scale  used 
for  utility  was  from  0  to  100,  where  a  utility  of  100  was  to 
be  assigned  to  a  value  within  a  criterion  which  would  most 
strongly  qualify  a  project  for  funding  (and  hence,  increase 
its  priority) .  Conversely,  a  utility  of  0  would  be  assigned 
to  a  value  within  a  criterion  which  would  most  strongly 


75 


disqualify  a  project  for  funding  (decrease  its  priority) . 

The  data  points  supplied  in  the  questionnaire  were  taken 
from  a  feasible  range  for  a  particular  criterion,  and 
therefore,  it  was  not  necessarily  expected  that  the 
endpoints  would  be  assigned  a  utility  of  0  or  100. 

The  third  questionnaire  only  requested  utility 
information  for  six  of  the  eight  criteria.  The  two  criteria 
which  were  omitted  were  'Technical  Feasibility',  and 
'Federal  Facility  Agreement'.  No  data  were  requested  for 
these  criteria  because  they  are  intended  to  be  ‘binary 
choice'  criteria.  Their  utility  values  will  either  be  0  if 
not  available  (or  not  present),  or  100  if  available  (or 
present) . 

The  cover  letter,  distribution  list,  and  questionnaire 
from  the  third  round  of  the  Delphi  are  included  at  Appendix 
H. 

Responses  to  the  third  Delphi  questionnaire  were 
received  from  six  of  the  twelve  experts.  Tables  3  through  8 
give  the  compiled  results  of  the  experts'  assignments  of 
utility  to  each  data  point  within  the  criteria. 

Development  of  the  Decision  Support  Model.  As 
discussed  in  Chapter  Three,  the  intent  of  the  Delphi 
procedure  was  to  collect  data  amenable  for  use  in  the 
construction  of  a  simple  additive  weighting  model  using  the 
criteria  and  utility  measures  elicited  from  the  experts, 
under  the  assumption  that  the  criteria  would  be 
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Table  3 

Utility  Results  for 

Threat  to  Human  Health  and  the  Environment 


Data  Point 

Expert  Number 

(DPM  Score) 

1 

2 

3 

4 

5 

6 

50 

100 

100 

80 

90 

100 

95 

40 

90 

95 

70 

75 

95 

90 

35 

70 

85 

55 

60 

90 

60 

30 

50 

75 

50 

30 

85 

50 

25 

30 

55 

35 

20 

80 

40 

20 

10 

35 

30 

10 

70 

15 

10 

0 

15 

10 

5 

50 

10 

Table  4 

Utility  Results  for 

Expected 

Reduction  of  Risk 

Expert  Number 

Data  Point 

1 

2 

3 

4 

5 

6 

90% 

100 

100 

90 

100 

100 

95 

75% 

100 

95 

75 

80 

90 

80 

60% 

100 

85 

60 

60 

80 

75 

50% 

80 

75 

50 

40 

70 

50 

40% 

60 

55 

40 

0 

50 

35 

30% 

50 

35 

20 

0 

30 

15 

20% 

20 

15 

10 

0 

10 

10 

preferentially  independent.  This  would  allow  the  model  to 
compute  a  score  for  a  project  by  convercing  each  of  that 
project's  attributes  (data  points  within  a  criterion)  into  a 
utility  value,  previously  defined  as  u4,  where  u  represents 


Table  5 

Utility  Results  for  Cost 


mm 

Data  Point 

1 

2 

3 

4 

5 

6 

$50,000 

100 

100 

90 

100 

94 

95 

$200,000 

90 

85 

85 

90 

90 

85 

$500,000 

85 

70 

80 

80 

85 

83 

$1,000,000 

75 

55 

70 

70 

80 

40 

$2,000,000 

50 

40 

60 

40 

70 

30 

$4,000,000 

20 

25 

50 

20 

50 

20 

$10,000,000 

10 

10 

30 

0 

30 

10 

Table 

6 

Utility  Results 

for  Community 

Acceptance 

Expat  Number 

Data  Point 

1 

2 

3 

4 

5 

6 

High 

100 

100 

50 

100 

90 

80 

Medium 

75 

65 

30 

80 

80 

80 

Low 

50 

25 

10 

0 

50 

60 

the  utility,  and  i  refers  to  a  specific  criterion,  and  then 
multiplying  this  utility  by  a  criterion-specific  weighting 
factor,  w4.  Thus,  the  score  for  a  project  would  be  the  sum 
of  each  of  these  criterion-specific  products,  w^. 

The  weighting  factors  were  determined  through  analysis 
of  the  responses  to  the  second  round  questionnaire. 


Table  7 

Utility  Results  for  Mission  Impact 


Expert  Number 


Data  Point 

1 

2 

3 

4 

5 

6 

Strong  Pos. 

100 

100 

30 

100 

90 

95 

Slight  Pos. 

20 

100 

20 

50 

75 

50 

No  Impact 

0 

100 

10 

40 

70 

40 

Slight  Neg. 

0 

60 

0 

30 

60 

25 

Strong  Neg 

0 

0 

0 

0 

20 

10 

Table  8 

Utility  Results  for  Socioeconomic/Political  Impact 


Expert  Number 


Data  Point 

1 

2 

3 

4 

5 

6 

Strong  Pos. 

100 

100 

30 

100 

90 

80 

Slight  Pos. 

40 

100 

20 

50 

80 

60 

No  Impact 

0 

100 

10 

40 

50 

35 

Slight  Neg. 

0 

60 

0 

10 

45 

20 

Strong  Neg. 

0 

0 

0 

0 

30 

5 

Analysis  of  variance  on  these  data  identified  a  number  of 
potential  and  probable  outliers  in  the  data  which  were 
removed  in  an  effort  to  decrease  the  overall  variance  of  the 
model  output .  The  median  values  for  each  criterion  were 
then  used  as  the  weight  for  that  criterion.  By  summing  the 
weights  to  arrive  at  a  total,  and  then  dividing  each  weight 
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by  the  total,  the  weighting  factors  were  normalized.  Table 
9  gives  the  median  weights  and  normalized  weighting  factors 
for  the  criteria. 


Table  9 

Median  Weights  and  Weighting  Factors  for  Criteria 


Criterion 

Median 

Weight 

Weighting 

Factor 

Threat  to  HH  &  E 

100.0 

0.183 

Expected  Reduction  of  Risk 

90.0 

0.164 

Technical  Feasibility 

85.0 

0.155 

Cost 

80.0 

0.146 

Federal  Facility  Agreement 

77.5 

0.142 

Community  Acceptance 

50.0 

0.091 

Mission  Impact 

50.0 

0.091 

Socioeconomic  Impacts 

15.0 

0.027 

Sum 

547.5 

To  extract  a  utility  function  from  the  responses 
provided  by  the  experts,  various  analyses  were  performed  on 
the  utility  data  for  each  criterion.  Again,  analysis  of 
variance  identified  a  number  of  potential  and  probable 
outliers  in  the  data  which  were  removed  prior  to  further 
analysis . 

Linear  regression  was  performed  on  the  utility  data  for 
the  'Threat  to  Human  Health  and  the  Environment ' ,  'Expected 
Reduction  of  Risk',  and  'Cost'  criteria.  Because  the  data 
for  cost  clearly  indicated  a  curving  trend,  as  can  be  seen 


80 


Expected  Value  of  Cost  (Uillions) 


Figure  3.  Plot  of  Expected  Value  of  Cost  vs.  Utility 

in  Figure  3,  these  data  were  transformed  using  a  cube  root 
function  prior  to  linear  regression.  This  analysis  resulted 
in  a  set  of  linear  equations  which  'best  fit'  the  data,  and 
defined  the  experts'  preference  structure.  The  resulting 
equations  were: 

UWWT  =  2.418 (DPM  Score)  -  18.726  (4) 

U„J8,  Mwcno*  =  1  •  214  ( %  Reduction)  -  5.541  (5) 

=  -0.492  (Cost)1'1  +  115.729  (6) 

Analysis  of  the  'goodness  of  fit'  for  these  equations  found 
the  following  statistics: 
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1)  The  coefficients  of  determination  (rJ)  for  ‘threat 
to  human  health  and  the  environment1,  ‘risk  reduction',  and 
'cost'  were  0.843,  0.875,  and  0.884,  respectively.  These 
statistics  indicate  most  of  the  variations  between  the 
criteria  and  utility  values  are  explained  by  the  regression 
equations . 

2)  The  t-statistics  for  'threat  to  human  health  and  the 
environment',  'risk  reduction',  and  ‘cost*  were  13.89, 

15.67,  and  -16.98,  respectively.  These  t-statistics 
indicate  that  the  variables  are  significantly  different  than 
zero,  which  implies  that  the  variable  is  significant  in 
explaining  the  resulting  utility  scores. 

3)  The  F-statistics  were  192.81,  245.42,  and  288.36  for 
'threat  to  human  health  and  the  environment',  'risk 
reduction',  and  'cost',  respectively.  For  these  simple, 
single-variable  regressions,  the  F-statistic  is  simply  the 
square  of  the  t-statistic,  and  therefore  implies  again  that 
the  variable  is  significant  in  explaining  the  resulting 
utility  scores. 


The  three  remaining  criteria  for  which  the  experts 
provided  data;  'Community  Acceptance',  'Mission  Impact*,  and 
' Socioeconomic/Political  Inpact ’  were  analyzed  for  the 
median  values  at  each  possible  data  point .  Since  these  data 
are  ordinal,  it  would  be  inappropriate  to  attempt  to  perform 
linear  regression.  Table  10  shows  the  median  values  for 
these  criteria. 

The  other  two  criteria,  'Technical  Feasibility',  and 
'Federal  Facility  Agreement',  required  no  analysis.  These 
criteria  are  binary,  and  because  this  is  a  deterministic 
model  (assumes  there  is  no  uncertainty  associated  with  these 
values)  the  resulting  utility  values  will  be  either  0  or 
100. 

Analysis  of  Variance.  A  statistical  evaluation  of  the 
resulting  model  can  be  performed  by  analyzing  the  variance 
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Table  10 

Median  Utility  Values  for  Ordinal  Criteria 


Community  Mission  Socioecooonac/PoliUcal 

Acceptance  _ Impact  Impact 


Data 

Point 

Median 

Utility 

Data 

Point 

Efflj I 

Data 

Point 

Median 

Utility 

High 

100.0 

Strong  Positive 

100.0 

Strong  Positive 

100.0 

Medium 

80.0 

Slight  Positive 

50.0 

Slight  Positive 

55.0 

Low 

37.5 

No  Impact 

40.0 

No  Impact 

350 

Slight  Negative 

27.5 

Slight  Negative 

10.0 

Strong  Negative 

0.0 

Strong  Negative 

0.0 

of  the  utility  function.  In  his  study  on  variability  in 
engineering  design,  Morrison  recited  an  equation  for 
deriving  the  propagated  variance  of  a  function  of  multiple 
random  variables  (1957:134).  The  variance  of  each  criterion 
score  is  expressed  as: 

Var( Criterion  Score(  )  =  ut*  •  Var(wt)  +  w/  •  Var(u_  )  (7) 

As  can  be  seen  in  this  equation,  the  variance  for  each 
criterion  is  a  multiplicative  function  of  the  variance  for 
weights  and  utilities.  Given  the  criteria  independence  as 
assumed  for  the  simple  additive  weighting  model,  the  total 
utility  function  variance  would  simply  be  the  sum  of  the 
individual  criterion  variances. 

While  this  research  is  interested  in  the  variation  of 
the  overall  utility  function,  this  total  variation  is 
dependent  on  each  criterion  variability  which  in  turn  is 
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dependent  on  the  individual  data  points.  For  example,  a  low 
DPM  score  has  a  different  variance  than  a  medium  or  high  DPM 
score.  The  selection  of  which  data  point  to  use  for  each 
criterion  greatly  influences  the  individual  criterion 
variance  and  thus  the  overall  utility  function  variance. 
Given  this  insight,  the  decision  was  made  to  calculate  the 
maximum  variance,  to  include  outliers,  that  would  result 
when  using  the  data  point  for  each  criterion  associated  with 
the  greatest  variance.  The  calculation  of  this  worst  case 
variance  is  shown  in  Appendix  I  and  equals  121.8  (standard 
deviation  of  11.04).  The  total  project  score  associated 
with  that  variation  was  51.9. 

Since  real  project  data  in  the  necessary  format  for 
scoring  were  not  available,  a  population  mean  and  variance 
cannot  be  calculated  for  a  comparison  with  this  variance. 
However,  one  would  suspect  that  the  calculated  maximum 
variance  for  project  scores  could  cast  considerable 
suspicion  on  the  final  ranking  produced.  Therefore,  using  a 
small  sample  of  real  data,  the  actual  variance  for  those 
data  in  lieu  of  the  maximum  variance,  and  the  assumption 
that  further  iterations  of  the  Delphi  would  have  eliminated 
outliers  (as  assumed  in  the  calculations  for  weights  and 
utilities),  the  model  was  further  tested  as  outlined  in  the 
following  section. 

Testing  the  Model.  In  order  to  perform  a  pilot  test  of 
the  model's  ability  to  rank  projects,  a  sample  of  actual 
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DERP  projects  was  selected  from  the  Air  Force  Materiel 
Command  FY94  funding  requests.  These  requests  are  submitted 
annually  by  the  individual  Air  Force  bases  to  the  MAJCOM 
CEVR.  Data  pertaining  to  these  requests  were  taken  from  the 
Military  Construction  Project  Data  Form  (DD  Form  1391)  and 
attached  project  summaries  included  in  each  submittal. 

Twelve  projects  were  selected  for  use  in  the  pilot  test. 
Table  11  gives  a  description  of  each  project. 

Table  11 

Projects  Included  in  Pilot  Test 


Project  DERA 


Number 

Location 

Description 

Priorit 

94-0203 

Hanscom  AFB 

Rl/FS  for  Unit  1  Petroleum  Spill 

2 

94-0816 

Hill  AFB 

IRA  for  RVMF  at  OU  3 

IA 

94-0843 

Hill  AFB 

RD/RA  for  OU  4 

1A 

94-1068 

Tinker  AFB 

IRA  Soils.  B-3001,  North  Fuel  Tank  Area 

IA 

94-1411 

Wright- Pat  ter  son  AFB 

Rl/FS  for  OU  9 

1A 

94-1709 

McClellan  AFB 

Ri/FS  for  OU  B 

IA 

94-1733 

McClellan  AFB 

RD/RA  for  GW  OU 

IA 

94-5008 

AFP  44 

PA/SI  for  Chip  Yards 

IA 

94-7003 

Newark  AFB 

RD/RA  for  Landfill  02 

2 

94-7022 

Rabins  AFB 

RD/RA  for  OU  1.  Drum  Removal 

IA 

94-7406 

Edwards  AFB 

RI/FS  for  OU  4.  Phillips  Lab 

IB 

94-7890 

Kelly  AFB 

IRA  at  Site  S^t  (ST-006) 

IA 

The  projects  were  selected  without  regard  to  location, 
DERA  priority,  or  any  of  the  individual  criteria  to  be  input 
in  the  model.  The  only  goal  in  the  selection  process  was  to 
obtain  a  representative  sample  of  projects  in  various  stages 
of  the  DERP  process  (i.e.,  studies,  designs,  and  remedial 
actions) . 
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Because  many  of  the  criteria  identified  by  the  experts 
as  being  important  in  the  DERP  prioritization  process  are 
not  currently  measured,  there  were  several  gaps  in  the 
project  summaries  where  data  required  for  analysis  was  not 
available.  Examples  of  this  include  ‘Community  Acceptance-, 
'Mission  Impact',  and  'Socioeconomic/Political  Impact'; 
these  types  of  information  are  not  currently  collected  or 
reported.  Where  these  data  gaps  existed,  assumptions  were 
made  based  upon  experience  and  any  situation-specific 
information  which  could  be  gathered  from  the  project 
summaries.  Table  12  gives  the  raw  data  gathered  from  these 
project  submittals. 

As  a  means  of  analyzing  the  variance  caused  by  the 
range  of  responses  which  were  received  from  the  experts,  two 
additional  sets  of  weighting  factors  and  utility  functions 
were  used  to  rank  the  projects.  These  weights  and  utilities 
were  taken  from  the  high  and  low  extremes  of  the  experts 
preference  structures.  By  multiplying  the  weighting  factors 
of  the  highest  expert  responses  by  the  highest  utility 
values,  and  similarly  multiplying  the  weighting  factors  of 
the  lowest  expert  responses  by  the  lowest  utility  values,  an 
'envelope'  would  be  created  about  the  median  weights  and 
'best-fit'  utility  functions.  Identical  rankings  of  the 
projects  by  each  method  would  indicate  that  despite  the 
variance  which  exists  between  individual  expert  opinions, 
the  difference  between  the  model  produced  using  the  median 
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Table  12 

Raw  Data  Gathered  from  DERP  Project  Submittals 


Raw  Data 

Hanecom 

Hill 

Hill 

Tinker 

WPAFB 

McClellan 

Criterion: 

Rl/FS 

IRA 

RD/RA 

IRA 

Rl/FS 

RD/RA 

Threat  to  HH  ft  E 

20 

34 

48 

41 

33 

28 

Expected  Reduction  of  Riak 

OH 

43H 

80H 

30H 

OH 

OH 

Technical  Feaaibibty 

Yea 

Yn 

Yea 

Yn 

Yn 

Yn 

CoM 

$22,190 

$607,000 

$3,300,000 

$1,330,000 

$1,104,000 

$3,318,000 

Federal  Facility  Agreement 

No 

Yn 

Yn 

Yn 

Yn 

Yn 

Comianity  Acceptance 

Medium 

High 

Medium 

High 

Medium 

Medium 

Minion  Impact 

None 

SI.  Neg 

SI.  Neg. 

None 

None 

SLPoa. 

Socioeconomic  Impact 

None 

SI.  Poa. 

SI.  Poa. 

SI.  Poa. 

None 

SLPoa. 

Raw  Date 

McClellan 

AFP  44 

Newark 

Robina 

Edwarda 

Kelly 

Criterion: 

RD/RA 

PA/SI 

RD/RA 

RD/RA 

Rl/FS 

IRA 

Threat  to  HH  ft  E 

38 

12 

34 

22 

20 

39 

Expectod  Reduction  of  Rak 

40% 

OH 

33H 

90% 

OH 

40% 

Technical  Feasibility 

No 

Yea 

Yn 

Yn 

Yn 

Yn 

Coat 

$2,000,000 

$73,000 

$4,300,000 

$83c:,ooo 

$4,378,304 

$3,400,000 

Federal  Facility  Agteemcrt 

Yn 

Yea 

No 

Yn 

No 

Yn 

Community  Acceptance 

Low 

High 

Medium 

High 

Medium 

High 

Minion  Impact 

None 

SI.  Poa. 

None 

None 

SLPoa. 

SLPoa. 

Socioeconomic  Impact 

SLPoa. 

None 

SL  Neg. 

None 

SI.  Poa. 

None 

values  for  weighting  factors  and  best-fit  utility  functions 
and  models  produced  using  the  most  extreme  values  for 
weights  and  utilities  is  not  significant.  Table  13  gives 
the  values  for  the  high  and  low  weights  and  the  normalized 
weighting  factors  associated  with  each  resulting  set  of 
criteria . 

The  utility  functions  were  again  calculated  using 
linear  regression  for  the  criteria  'Threat  to  Human  Health 
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Table  13 

High  and  Low  Weights  and  Weighting  Factors 


Criterion 

High 

Weight 

Threat  to  HH  &  E 

100.0 

Expected  Reduction  of  Risk 

95.0 

Technical  Feasibility 

98.0 

Cost 

97.0 

Federal  Facility  Agreement 

93.0 

Community  Acceptance 

85.0 

Mission  Impact 

80.0 

Socioeconomic  Impacts 

80.0 

Sum 

728.0 

Normalized 

Weighting  Low 

Factor  Weight 

Normalized 

Weighting 

Factor 

100.0 

0.256 

0.130 

80.0 

0.205 

0.135 

70.0 

0.179 

0.133 

60.0 

0.154 

0.128 

30.0 

0.077 

0.117 

30.0 

0.077 

0.110 

20.0 

0.051 

0.110 

0.0 

0.000 

1.000 

390.0 

1.000 

and  the  Environment',  'Expected  Reduction  of  Risk',  and 
'Cost',  as  shown  below  for  the  high  utility  values: 


UTO>»t  =  2.357  (DPM  Score)  -  2.857  (8) 

UBI„ =  1.197  (%  Reduction)  +  6.141  (9) 

Uco„  =  -0.394  (Cost ) 1/5  +  115.795  (10) 

and  the  low  utility  values: 

Umrai  =  2.262 (DPM  Score)  -  27.143  (11) 

U««p*ducti»=  1.199  (%  Reduction)  -  16.102  (12) 

Uco„  =  -0.560  (Cost)1'5  +  113.401  (13) 
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Analysis  of  the  'goodness  of  fit'  for  these  equations  found 
the  following  statistics: 

1)  For  the  high  utility  values,  the  coefficients  of 
determination  (r1)  for  'threat  to  human  health  and  the 
environment',  'risk  reduction',  and  'cost'  were  0.885, 

0.892,  and  0.987,  respectively.  For  the  low  utility  values, 
the  coefficients  of  determination  (rJ)  for  'threat  to  human 
health  and  the  environment',  'risk  reduction',  and  'cost' 
were  0.939,  0.986,  and  0.948,  respectively. 

2)  For  the  high  utility  values,  the  t-statistics  for 
'threat  to  human  health  and  the  environment',  'risk 
reduction',  and  'cost'  were  6.20,  6.42,  and  -19.09, 
respectively.  For  the  low  utility  values,  the  t-statistics 
for  ’threat  to  human  health  and  the  environment',  'risk 
reduction',  and  'cost'  were  8.77,  18.83,  and  -9.54, 
respectively. 

3)  For  the  high  utility  values,  the  F-statistics  were 
38.44,  41.23,  and  364.46  for  ‘threat  to  human  health  and  the 
environment',  'risk  reduction',  and  'cost',  respectively. 

For  the  low  utility  values,  the  F-statistics  were  76.87, 
354.74,  and  91.07  for  'threat  to  human  health  and  the 
environment',  'risk  reduction',  and  'cost',  respectively. 


The  three  ordinal  criteria;  'Community  Acceptance', 
'Mission  Impact',  and  'Socioeconomic/Political  Inpact'  were 
analyzed  at  the  high  and  low  data  points  as  well.  Table  14 
gives  the  utilities  for  each  of  the  data  points  within  these 
criteria. 

The  values  of  the  two  binary  criteria;  'Technical 
Feasibility',  and  'Federal  Facility  Agreement'  did  not 
change  for  this  portion  of  the  analysis,  and  still  assumed 
values  of  0  or  100. 

Results  of  the  Decision  Support  Model  Pilot  Test.  By 
using  the  data  provided  by  the  AFMC/CEVR  projects  as  input 


89 


Table  14 

High  and  Low  Utility  Values  for  Ordinal  Criteria 


High  Utility  Values: 
Community 


Mission 

Impact 


Socioecononac/PoliUcal 


Data 

Point 

u\mm 

ESSE 

Data 

Point 

mm 

in 

Data 

Point 

High 

Utility 

High 

100.0 

Strong  Positive 

100.0 

Strong  Positive 

100.0 

Medium 

80.0 

Slight  Positive 

75.0 

Slight  Positive 

80.0 

Low 

60.0 

No  Impact 

70.0 

No  Impact 

50.0 

Slight  Negative 

60.0 

Slight  Negative 

45.0 

Strong  Negative 

20.0 

Strong  Negative 

5.0 

Low  Utility  Values: 


Community  Mission  Socioeconomic/Political 

Acceptance  _ Impact  Impact 


Data 

Point 

BBS 

Data 

Point 

H 

ES31 

Data 

Point 

Low 

Utility 

High 

80.0 

Strong  Positive 

90.0 

Strong  Positive 

80.0 

Medium 

65.0 

Slight  Positive 

20.0 

Slight  Positive 

40.0 

Low 

0.0 

No  Impact 

0.0 

No  Impact 

0.0 

Slight  Negative 

0.0 

Slight  Negative 

0.0 

Strong  Negative 

0.0 

Strong  Negative 

0.0 

to  the  median,  high,  and  low  weight  and  utility  models, 
three  sets  of  scores  were  produced.  From  these  scores, 
three  rankings  of  the  projects  were  assembled.  Table  15 
shows  the  scores  and  rankings  for  each  project  by  each 
version  of  the  model. 

Inspection  of  the  correlation  between  these  rankings 
reveals  similar  results  by  all  three  methods.  Only  slight 
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Table  15 

Scores  and  Rankings  of  Projects 


Location 

Action 

High 

Score 

Median 

Score 

Low 

Score 

High 

Rank 

Median 

Rank 

Low 

Rank 

Hill 

REVRA 

83.198 

79.802 

72.190 

1 

1 

1 

Hill 

IRA 

82.751 

73.278 

62.425 

2 

2 

2 

Tinker 

IRA 

82.421 

72.672 

60.562 

3 

3 

3 

Kelly 

ERA 

78.491 

71.288 

59.446 

4 

4 

4 

Robin 

RD/RA 

75.658 

68.410 

56.375 

5 

5 

5 

WPAFB 

RI/FS 

70.550 

63.107 

52.476 

6 

6 

6 

AFP  44 

PA/SI 

69.241 

60.076 

46.640 

8 

7 

8 

McClellan 

RI/FS 

69.583 

57.961 

45.241 

7 

8 

9 

Newark 

REVRA 

62.199 

53.399 

49.094 

10 

9 

7 

McClellan 

REVRA 

64.017 

51.037 

35.889 

9 

10 

11 

Hanscom 

RI/FS 

55.151 

45.341 

39.781 

11 

11 

10 

Edwards 

RI/FS 

53.467 

39.219 

31.705 

12 

12 

12 

variations  in  the  rankings  occur  in  the  lower-priority 
projects . 

Analysis  of  the  variance  among  the  rankings  was 
performed  using  two  methods  proposed  by  Saaty:  the  root 
mean  square  (RMS)  method,  and  the  median  absolute  deviation 
about  the  median  (MAD)  method.  Each  of  these  methods 
provide  a  procedure  to  determine  the  accuracy  of  two  sets  of 
numbers  in  absolute  terms.  (Saaty,  1980:37)  Given  two  sets 
of  numbers  a,,...,  a.,  and  b,,...,  b„,  the  general  form  of  the 
RMS  equation  is: 


rms  = 

1  n  i.i 
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The  general  form  of  the  MAD  equation  is: 


MAD  =  mediant  I  (a,  -  b,)  -  median  (a,  -  b.)  1}  (14) 

By  applying  these  methods  to  the  rankings  determined 
previously  by  the  three  methods,  two  measures  of  the 
closeness  of  the  results  are  determined.  Table  16  shows  the 
RMS  calculations  for  the  rankings,  and  Table  17  shows  the 
MAD  calculations  for  the  rankings. 

Saaty  suggests  that  the  significance  of  both  the  RMS 
and  the  MAD  methods  may  be  determined  by  dividing  their 
values  by  the  average  value  of  the  vector  components,  1/n, 
where  n  is  the  number  of  components.  A  ratio  of  0.1  or  less 
from  either  or  both  methods  indicates  that  the  vectors  being 
compared  are  nearly  the  same.  (Saaty,  1980:39) 

In  the  RMS  cases,  these  values  are  6.6882  for  the  high 
value  and  median  value  comparison,  and  9.7980  for  the  median 
value  and  low  value  comparison.  In  the  MAD  cases,  however, 
these  values  are  zero  for  the  high-median  comparison,  as 
well  as  the  median-low  comparison.  This  indicates  that  the 
vectors  are  nearly  the  same  since  only  one  of  the  methods  is 
required  to  produce  a  ratio  below  0.1.  Therefore,  as  noted 
above,  although  differences  exist  between  individual 
opinions,  the  conglomerate  of  median  weights  and  utilities 
produce  a  model  which  is  not  significantly  different  fror.i  a 
model  produced  using  the  extreme  weights  and  utilities. 
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Table  16 

Root  Mean  Square  Calculations  for  Rankings 


RMS  Calculation  for  High  vs.  Median  Rankings: 
High  Median 


Ranking* 

(a) 

Rankings 

(b) 

(a  -  b) 

(a  -  b)2 

1 

1 

0 

0 

2 

2 

0 

0 

3 

3 

0 

0 

4 

4 

0 

0 

5 

5 

0 

0 

6 

6 

0 

0 

8 

7 

1 

1 

7 

8 

-1 

1 

10 

9 

1 

1 

9 

10 

-1 

1 

11 

11 

0 

0 

12 

12 

0 

0 

SSE  = 

4 

RMS  * 

0.57735 

RMS  Calculation  for  Median  v*  Low  Rankings: 

Median  Low 

Rankings  Rankings 

<SL _ £) _ (a-b) _ 

(a  -  b)2 

1 

1 

0 

0 

2 

2 

0 

0 

3 

3 

0 

0 

4 

4 

0 

0 

5 

5 

0 

0 

6 

6 

0 

r 

7 

8 

-1 

l 

8 

9 

-1 

l 

9 

7 

2 

4 

10 

11 

-1 

1 

11 

10 

1 

1 

12 

12 

0 

0 

SSE  =  8 

RMS  =  0.816497 


Table  17 

Median  Absolute  Deviation  About  the  Median 
Calculations  for  Rankings 


MADCakdaban  fcr  w.  Medwi  Raring* 
Hgfc  Medan 


Ranking* 

Ranking* 

(a) 

(b) 

(a-b)  Ka-b)- 

medarKa  -  Dj 

1 

1 

0 
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This  result  implies  that  the  median  weights  and  utilities 
adequately  represent  the  experts'  opinions,  taken  as  a 
whole . 

Further  sensitivity  analysis  of  the  data  was  performed 
to  examine  the  effects  of  the  outliers  on  the  performance  of 
the  model.  Again,  three  models  were  developed  using  the 
median  data  and  the  extreme  high  and  low  data.  For  these 
models,  however,  the  outliers,  which  were  previously  omitted 
from  the  analysis,  were  included.  This  situation  resulted 
in  models  which  produced  significantly  different  rankings. 
Therefore,  it  should  be  noted  that  the  validity  of  the  final 
model  is  dependent  upon  the  degree  of  consensus  reached 
among  the  experts  in  that  the  outliers  will  have  a 
significant  effect  on  the  ordering  of  projects  in  the  final 
model . 

Analysis  of  Pat.a.  from  the,..MAVT  Erocetiure 

As  noted  in  Chapter  3,  there  are  five  steps  to  the 
value-function  measurement  task.  The  first  step  is  to 
familiarize  the  decision  maker  with  the  concepts  and 
techniques  of  MAUT.  The  second  is  to  identify  the  form  of 
the  utility  function.  The  form  is  dependent  on  the 
relationships  between  the  criteria  of  interest,  which  can  be 
determined  by  the  responses  to  the  questionnaire. 

The  responses  to  questions  i-6  were  used  to  test  for 
preferential  independence.  In  these  six  questions,  the 
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decision  maker  was  given  a  hypothetical  situation  with  two 
projec_  alternatives.  The  odd  numbered  questions 
established  the  relative  importance  among  two  of  the 
criteria  while  holding  the  third  criterion  fixed.  Even 
numbered  questions  were  used  to  alter  the  fixed  criterion  to 
determine  if  the  relative  importance  established  in  the 
preceding  question  changed.  In  all  cases,  the  relationship 
between  the  two  criteria  of  interest  remained  the  same, 
establishing  preferential  independence.  That  is,  the  value 
trade-off  between  every  pair  of  attributes  was  not  affected 
by  a  third  attribute  (Chan,  1993:41).  Since  all  attributes 
were  tested  against  one  another,  and  all  proved  to  be 
preferentially  independent,  the  entire  set  of  attributes  is 
mutually  preferentially  independent  (Winston,  1991:730). 

Another  important  relationship  to  determine  is  mutual 
utility  independence.  If  the  relative  utility  of  each 
criterion  remains  the  same  regardless  of  the  value  of  other 
criteria,  the  criteria  are  said  to  be  utility  independent. 
Mutual  utility  independence  implies  that  both  a  criteria  set 
and  its  complement  are  utility  independent  (Chan, 
1993:42,43).  Questions  7-9  established  that  the  criteria  of 
risk,  cost,  and  Major  Command  (MAJCOM)  priority  are  not 
utility  independent.  With  the  exception  of  MAJCOM  priority, 
the  utility  associated  with  a  criterion  changed  as  the  other 
two  criteria  changed.  With  MAJCOM  priority,  the  decision 
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maker  indicated  that  he  had  no  utility  for  this  criterion 
regardless  of  the  value  or  DPM  scores  or  costs. 

Establishing  mutual  preferential  independence  implies 
that  the  resulting  function  will  be  an  additive  value 
function  (Winston,  1991:730).  This  is  consistent  with  the 
finding  that  mutual  utility  independence,  which  is  required 
for  multiplicative  utility  models,  was  absent. 

Unfortunately,  the  fact  that  mutual  utility  independence  was 
not  established  also  limits  the  conclusions  one  can  draw 
when  quantifying  the  intensity  of  choice  which  is  required 
for  cardinal  utility  functions  (Chan,  1993:41,43). 

The  third  step  of  the  MAUT  process,  the  overall 
measurement  of  utility,  is  based  on  the  premise  that  "the 
utility  of  an  alternative  is  [equal  to]  the  sum  of  the 
utility  of  each  of  the  possible  outcomes"  (Chan,  1993:34). 

To  determine  the  univariate  value  functions,  questions  10 
and  11  tested  the  decision  maker’s  utility  for  risk  and 
cost.  The  utility  for  MAJCOM  priority  was  pre-established 
as  discussed  in  Chapter  3 . 

In  both  questions  10  and  11,  lotteries  are  used  to 
provide  a  measure  of  utility  for  use  in  the  multicriteria 
decision  model  (MCDM) .  Figures  4  and  5  show  plots  of  the 
responses  to  these  two  questions.  On  the  x-axis,  the 
expected  value  of  the  lottery  choice  is  plotted.  On  the  y- 
axis,  the  certainty  equivalent  is  plotted.  The  certainty 
equivalent  is  defined  as  an  amount  such  that  the  decision 
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maker  is  indifferent  between  it  with  certainty  and  the 
expected  value  of  the  lottery  amount  (Chan,  1993:36). 

As  shown  by  the  curve  in  Figure  4,  the  decision  maker 
is  risk  averse  for  the  utility  associated  with  DPM  score. 
Being  conservative,  he  values  the  reduction  in  DPM  scores 
associated  with  the  'certain  outcome'  over  the  higher 
'expected'  reduction  of  risk  associated  with  the  'uncertain 
outcome' .  His  utility  can  be  represented  in  algebraic  form 
as : 

Utility**  =  2.07  (DPM)0"  (15) 

The  coefficient  of  determination  (rJ  statistic)  for  this 
regression  equation  is  0.998,  indicating  that  virtually  all 
of  the  variance  between  the  criterion  data  points  and  the 
utility  data  points  is  explained  by  the  equation. 

On  the  other  hand,  the  decision  maker  is  risk  neutral 
on  the  utility  associated  with  costs.  In  all  cases,  he 
would  be  willing  to  pay  someone  else  exactly  the  expected 
value  of  the  cost  of  corrective  actions  at  a  site  for 
assuming  the  risk.  Said  differently,  he  would  be 
indifferent  to  assuming  the  full  cost  of  corrective  actions 
at  a  site  vice  paying  someone  else  the  expected  costs  and 
requiring  them  to  assume  responsibility  for  the  full  cost. 

The  bottom  line  when  using  a  questionnaire  is  to  arrive 
at  an  overall  value  function.  In  order  to  represent  the 
various  alternatives  in  an  overall  value  function,  scaling 
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functions  and  the  relative  weights  for  each  criterion  must 
be  determined.  Scaling  involves  the  transformation  of  data 
to  a  value  between  0  and  1.  Simple  linear  transformations 
can  be  performed  by  dividing  the  outcome  by  its  maximum 
value.  In  the  case  of  costs,  the  transformation  is 
accomplished  by  taking  the  inverse  of  the  outcomes  (e.g., 
1/outcome) .  The  advantage  to  linear  transformations  is  that 
they  maintain  the  relative  order  of  magnitude  of  outcomes 
(Hwang,  1981 : 30)  . 

A  scaled  utility  function  for  MAJCOM  priority  was  given 
in  equation  (3)  in  Chapter  3  and  is  rephrased  here: 

Value„  =  1  -  (priority  /  total  #  of  projects)  (16) 

This  function  converts  all  priorities  to  a  scale  of  0  to  1, 
with  the  highest  priority  project  having  a  value  almost 
equal  to  1 . 

A  scaling  function  for  risk  is  also  required.  For  this 
function,  t  is  assumed  that  maximum  utility  is  associated 
with  any  DPM  score  over  50  and  that  minimum  utility  is 
associated  with  any  DPM  score  of  0.  In  actuality,  the  lower 
bound  for  DPM  should  be  determined  by  the  future  land  use  of 
the  site  (Owendoff ,  1993b) ;  however,  for  the  sake  of 
simplicity,  we  will  assume  the  lower  bound  to  be  0.  The 
ratio  needed  to  convert  the  DPM  score  to  a  value  in  the 
range  of  0  to  1  is  simply  the  inverse  of  the  maximum 
expected  value,  or  1/50.  Combining  this  scaling  ratio  with 
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the  utility  function  given  earlier  yields  a  scaled  utility 
function  of: 

Value,*.  =  0.0414 (DPM)“"  (17) 

The  scaled  utility  function  for  cost  was  similarly 
determined.  For  cost,  the  maximum  value  used  for  utility 
comparisons  was  $10  million.  However,  $10  million  may  not 
be  the  upper  limit  on  project  costs.  Since  the  utility  for 
cost  was  linearly  related  to  the  expected  value,  there  is  no 
reason  to  limit  the  utility  function  to  $10  million. 
Therefore,  the  scaled  utility  function  becomes: 

Value,**,  =  1  /  maximum  project  cost  (18) 

The  weights  for  each  of  the  criteria  can  be  determined 
algebraically  from  the  responses  in  the  questionnaire  and  a 
simplification  to  the  following  relationship  for  a  three- 
variable  multiplicative  value  function: 

Value  (y,,y,,y,)  =  w,v(yt)  +  wav(ya)  +  w/vty,)  + 

+  k[w,wav(y,)v(ya)  +  w.WjVty.)  v(ya)  + 

+  wawav(ya) v(y3)  ]  +  kJ Iw.w^vfy,) v(ya) v(y3)  ]  (19) 

where  w,  refers  to  the  weights,  k  is  a  multiplicative 
scaling  constant,  and  the  subscripts  1,  2,  and  3  refer  to 
DPM  scores,  MAJCOM  priority,  and  cost  respectively  (Chan, 
1993:33).  Since  the  decision  maker  noted  at  the  end  of 
Section  A  that  his  utility  for  MAJCOM  priority  was  zero 
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(i.e.,  v(yj=0)  and  confirmed  this  in  his  responses  to 
questions  3,  4,  and  8,  and  since  the  form  of  the  function 
was  determined  to  be  simple  additive  weighting  (i.e.,  k=0); 
the  equation  simplifies  to: 

Value  (y,,y,,y,)  =  w,v(y,)  +  w,v(yj  (20) 

The  relationship  between  DPM  scores  and  costs  can  be 
calculated  from  the  responses  to  questions  5,  6,  and  14.  In 
questions  5  and  6,  the  DPM  score  varies  from  30  to  15 
between  the  two  hypothetical  projects.  The  project  cost  at 
the  higher  risk  site  was  given  as  $500,000.  The  decision 
maker  was  asked  to  chose  a  project  cost  for  the  second  site 
that  would  make  him  indifferent  between  the  two  projects. 

The  decision  maker  responded  by  saying  he  would  always  chose 
the  higher  risk  site  regardless  of  cost  differences.  This 
response  indicates  that  although  the  decision  maker  has 
utility  associated  with  the  cost  criterion,  the  weight  he 
would  assign  cost  in  relation  to  risk  is  low  (see  Appendix 
J) .  This  response  was  verified  in  a  follow-on  interview 
with  the  decision  maker  after  the  decision  maker  was 
informed  of  the  implications  (Edwards,  1993a) .  These 
results  are  also  consistent  with  the  responses  to  questions 
7,  9,  14,  and  15.  (It  should  be  noted  that  the  response  to 
7.1  raises  suspicion  that  the  cost  criterion  does  enter  in 
to  the  overall  value  function;  however,  the  decision  maker 
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indicated  that  a  difference  of  10  in  DPM  scores  makes  the 
comparison  of  risk  less  certain.) 

Given  that  the  weights  and  utilities  associated  with 
the  cost  and  MAJCOM  priority  variables  are  zero,  the  final 
value  function  simplifies  to  the  univariate  risk  function. 
Since  the  decision  maker  indicated  that  small  differences  in 
DPM  may  not  have  meaning,  the  requirement  for  a  uniform 
preference  intensity  for  a  cardinal  scale  is  violated  and 
the  ranking  is  strictly  an  ordinal  ranking  on  DPM  scores. 
Again,  this  is  consistent  with  the  fact  that  preferential 
independence  was  established  but  not  mutual  utility 
independence . 

The  obvious  question  at  this  point  in  the  research  is: 
“Why  does  the  model  only  use  one  criterion  to  rank 
projects?-  The  explanation  could  be  with  the  criteria 
determined  from  the  pre-questionnaire,  the  metrics  used  to 
represent  the  criteria,  or  the  questionnaire. 

The  questionnaire  results  appear  to  be  consistent. 
Questions  that  made  a  pairwise  comparison  among  similar 
criteria  were  answered  in  similar  fashion.  Additionally, 
the  response  to  the  'control'  question,  question  15,  rated 
DPM  score  as  most  important  and  MAJCOM  priority  as  least 
important.  Finally,  the  proposed  prioritization  scheme  the 
decision  maker  sketched  on  the  bottom  of  the  questionnaire 
uses  only  DPM  scores  to  choose  among  similar  categories  of 
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projects.  Neither  cost,  priority,  nor  any  of  the  other 
criteria  proposed  by  the  experts  are  present  in  this  scheme. 

To  gain  further  insight  into  these  results,  a  brief, 
informal,  post -questionnaire  interview  was  conducted  with 
the  decision  maker.  Col.  Owendoff.  He  indicated  risk  was  of 
paramount  importance  and  cost  was  secondary  to  risk.  His 
rationale  was  that  the  Air  Force  must  demonstrate  to 
Congress  that  it  is  taking  on  all  projects  above  a  given 
level  of  risk.  He  also  indicated  that  while  the  MAJCOMs 
currently  play  an  important  role  in  allocating  funds  and  are 
closer  to  the  problems  than  the  Air  Staff,  the  MAJCOM 
priority  criterion  was  not  'concrete'  enough.  The 
stakeholders  in  the  IRP  need  criteria  that  can  be  applied 
consistently  across  commands  and  have  a  less  subjective 
basis  for  their  use.  (Owendoff,  1993c)  In  addition  to  the 
post -questionnaire  interview  with  Col.  Owendoff,  a  final 
copy  of  the  analysis  and  the  inferences  drawn  were  provided 
to  Mr.  Edwards.  Mr.  Edwards,  like  Col.  Owendoff,  also 
indicated  concurrence  with  the  results  (Edwards,  1993b) . 

Comparison  of  the  Delphi  and  MAUT  Procedures 

The  results  of  the  two  procedures  can  be  analyzed  in  a 
manner  similar  to  that  of  the  results  of  the  Delphi,  that 
is,  a  direct  comparison  of  the  project  rankings  of  the  two 
methods.  Table  18  shows  the  twelve  AFMC  projects  as  they 
were  previously  ranked  by  the  Delphi  model  (median  weights 
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Table  18 

Comparison  of  Delphi  and  MAUT  Project  Rankings 
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and  utilities) ,  and  compares  them  to  the  project  rankings  by 
the  MAUT  model  (DPM  score) .  This  comparison  also  shows  the 
calculations  for  root  mean  square  (RMS)  and  median  absolute 
deviation  about  the  median  (MAD) .  By  dividing  the  RMS  and 
MAD  vectors  by  1/n,  their  significance  may  be  determined. 

In  this  case,  the  RMS  analysis  yields  a  value  of 

RMS/ (1/n)  =  (3.02765) / (1/12)  =  36.3318  (21) 

and  the  MAD  analysis  yields  a  value  of 

MAD/ (1/n)  =  (1) / (1/12)  =  12  (22) 

Since  neither  of  these  values  is  below  0.1,  it  cannot 
be  stated  that  the  rankings  of  the  two  models  are  nearly  the 
same,  and  therefore,  the  prioritization  models  themselves 
must  be  significantly  different. 

Conclusion 

The  results  of  the  above  comparison  indicate  that  the 
DERP  experts  and  the  decision  maker  may  have  different 
perceptions  of  the  problem  and  different  objectives  from  the 
prioritization  process.  Possible  explanations  for  this  can 
be  derived  by  investigating  the  goals  of  the  experts  and  the 
decision  maker.  The  experts,  having  technical  knowledge  of 
the  issues  involved  with  the  DERP,  may  be  more  concerned 
with  the  attributes  involved  in  the  projects.  They  do  not 
have  the  responsibility  of  deciding  which  projects  get 
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funding.  Viewed  this  way,  the  experts  can  be  seen  as 
examining  the  problem  in  a  prospective  manner.  They  will 
tend  to  be  more  concerned  with  investigating  all  relevant 
attributes  of  projects  when  prioritizing  those  projects,  and 
less  concerned  with  the  outcomes  of  the  process.  A  decision 
maker,  on  the  other  hand,  may  view  the  problem  differently. 

A  decision  maker  is  required  to  provide  outcomes  and  to  make 
decisions,  and  therefore  may  have  a  more  retrospective  view 
of  the  problem.  This  view  may  cause  the  decision  maker  to 
limit  the  number  of  project  attributes  which  are  considered 
when  prioritizing  projects  for  funding. 

Another  possibility  for  the  difference  involves  the 
number  of  inputs  into  the  model .  The  Delphi  procedure 
employed  12  experts  who  all  had  input  into  the  criteria 
used.  As  noted  earlier,  this  is  likely  to  expand  the  range 
of  issues  considered.  The  MAUT  procedure  employed  just  one 
decision  maker.  If  the  decision  maker  didn't  feel  a 
particular  criterion  was  valid,  it  was  eliminated.  In  the 
Delphi,  differences  of  opinion  on  the  validity  of  criteria 
resulted  in  variance,  but  the  criteria  were  still  included. 
Despite  these  differences,  both  methods  produced  usable  data 
for  the  resulting  decision  support  system.  The  usefulness 
of  the  resulting  decision  support  systems  will  be  discussed 
in  greater  detail  in  the  next  chapter. 
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)L* _ Conclusions  and  Recommendations 


The  two  proposed  multicriteria  decision  models  (MCDMs ) 
arrived  at  in  Chapter  4  both  provide  an  alternative  to  the 
research  problem  posed  in  Chapter  1,  development  of  a  means 
to  differentiate  Defense  Environmental  Restoration  Program 
(DERP)  projects.  While  the  two  methods  do  not  arrive  at  a 
common  solution,  they  both  satisfy  the  key  program 
objectives  and  respond  to  the  overall  needs.  Either  option 
could  be  used  in  a  decision  support  system  (DSS)  to 
initially  rank  projects. 

Program , Objectives 

Before  developing  an  MCDM,  the  program  objectives  were 
researched.  As  determined  from  the  literature  review,  che 
overriding  goal  in  the  statutes.  Environmental  Protection 
Agency  (EPA)  regulations,  and  Department  of  Defense  (DoD) 
regulations  is  to  protect  the  public  health.  Reducing  site 
risk,  attacking  the  worst  sites  first,  controlling  immediate 
threats,  and  a  bias  toward  remedial  action  vice  studies  all 
are  based  on  the  protection  of  public  health. 

The  regulatory  objectives  also  matched  DoD  objectives 
in  regards  to  cleaning  up  contaminated  sites.  Both  sought 
quick,  effective  cleanup  strategies  with  an  end  goal  of 
meeting  applicable  or  relevant  and  appropriate  requirements 
(ARARs)  and  site  closure.  The  overall  objective  with 
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regards  to  cost-ef _ectiveness  were  also  similar; 
unfortunately,  neither  program  really  defined  a  measurable 
goal  for  this  objective.  Finally,  both  the  regulators  and 
DoD  indicated  streamlining  procedures,  meeting  statutory 
deadlines  and  requirements,  and  complying  with  court  orders 
or  other  forms  of  agreements  were  important. 

Other  objectives  mentioned  in  the  literature  were 
either  agency-specific  or  were  less  closely  matched.  For 
exa^ole,  the  EPA  objective  of  encouraging  innovation  doesn't 
coincide  with  the  DoD  goal  of  standardizing  cleanup  methods. 
It  does,  however,  match  the  DoD  policy  of  funding  research 
and  development.  From  an  investment  perspective,  research 
into  innovative  solutions  could  result  in  more  efficient  and 
effective  solutions  for  future  sites.  From  a  practical 
standpoint,  however,  EPA's  desire  to  test  innovative 
solutions  doesn't  coincide  with  DoD's  realization  that 
innovation  involves  risking  limited  funds,  takes  time,  can 
be  difficult  to  implement,  and  may  not  show  progress  in 
cleaning  up  the  environment . 

Model  Development 

Criteria .  As  noted  in  the  methodology  section,  the 
criteria  selected  for  use  in  the  model  is  one  of  the  most 
critical  elements.  Both  the  Delphi  and  multiattribute 
utility  theory  (MAUT)  procedures  relied  on  the  criteria 
mentioned  in  the  literature  as  a  starting  point  for 


109 


developing  a  model .  The  Delphi  procedure  sought  the  opinion 
of  experts  to  refine  the  list,  while  the  MAUT  procedure 
relied  on  a  pre-questionnaire  interview  and  the  opinion  of 
the  researchers  to  narrow  the  criteria  to  a  suitable  number. 

The  criterion  most  frequently  mentioned  in  the 
literature  and  most  heavily  weighted  in  both  models  was 
risk.  Risk,  or  the  threat  to  the  public  health  or 
environment,  was  represented  in  both  models  by  the  Defense 
Priority  Model  (DPM)  score.  Other  possible  criteria  related 
to  risk  include  risk  reduction;  the  reduction  in  toxicity, 
mobility,  or  volume  of  contamination;  and  short-term 
effectiveness.  Of  these,  only  the  expected  reduction  of 
risk  as  measured  by  the  expected  reduction  in  DPM  score  was 
included.  It  was  the  second  highest  weighted  factor  in  the 
Delphi  model,  but  it  was  not  included  in  the  MAUT  MCDM. 

Many  of  the  other  criteria  mentioned  in  literature 
involved  compliance.  Compliance  with  regulatory  agreement 
schedules,  with  statutory  deadlines  and  requirements,  with 
ARARs,  or  with  DoD  or  Air  Force  goals  all  were  possible 
criteria  for  use  in  the  models.  The  Delphi  model  included 
federal  facility  agreement  compliance,  a  binary  criterion, 
as  the  fifth  most  important  criterion.  The  MAUT  procedure 
grouped  all  these  measures  together  in  a  subjective  'MAJCOM 
priority'  criterion,  but  the  decision  maker's  responses  to 
the  questionnaire  negated  its  use  in  the  model. 
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Whether  or  not  the  proposed  project  is  technically  or 
contractually  feasible  is  also  important  to  the  evaluation 
and  prioritization  of  projects.  Technical  feasibility,  as 
measured  by  the  availability  of  generally  accepted  methods 
for  remediation,  was  the  third  highest  weighted  criterion  in 
the  Delphi  MCDM.  In  the  MAUT  model,  technical  feasibility 
was  incorporated  into  MAJCOM  priority  (which  was 
subsequently  dismissed  through  analysis  of  the  decision 
maker's  responses). 

Cost,  like  risk,  was  a  criterion  in  both  models.  Since 
the  overall  cost  of  completing  the  cleanup  of  a  site  is  very 
difficult  to  assess  and  the  estimates  for  later  stages 
depend  on  the  results  of  studies  in  the  earlier  stages,  the 
current  year  project  cost  was  used  as  the  only  measure  of 
cost.  Cost  was  identified  as  the  fourth  most  important 
criterion  to  be  considered  in  the  Delphi  model.  The 
decision  maker  for  the  MAUT  procedure  assigned  a  utility  to 
cost,  but  his  relative  priority  for  the  risk  criterion 
completely  dominated  cost . 

Community  acceptance  was  the  key  parameter  in  the  most 
recent  study  on  IRP  prioritization,  the  Keystone  report. 

For  this  research,  the  Delphi  experts  gave  community 
acceptance  some  consideration;  however,  the  measurement  is  a 
subjective  measure  by  the  decision  makers  and  is  tied  for 
next-to-last  in  importance.  Again,  for  the  MAUT  procedure, 
community  acceptance  and  concurrence  of  the  technical  review 
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committee  was  included  in  MAJCOM  priority  and  subsequently 
dismissed.  State  and  federal  EPA  acceptance,  while  a  valid 
criterion,  was  not  considered  in  either  model. 

Finally,  the  impact  of  a  project  on  the  mission  of  the 
installation  or  the  socioeconomic  impacts  of  the  project  on 
the  community  were  mentioned  in  the  literature  for  possible 
consideration.  Both  criteria  were  included  by  the  experts 
in  the  Delphi  model,  although  mission  inpact  was  considered 
next-to-last  in  importance  and  socioeconomic  inpact 
contributed  less  than  3%  to  the  overall  score  of  a  project. 
Both  factors  were  also  included  in  the  MAUT  model  under  the 
MAJCOM  priority  criterion,  and  subsequently  dismissed. 

Assumptions .  Both  the  Delphi  and  MAUT  MCDMs  assume 
installation  level  personnel,  along  with  the  possibility  of 
a  technical  review  committee  comprised  of  regulatory  and 
community  representatives,  submit  appropriate  projects  for 
funding  consideration.  By  making  this  assumption,  the  MCDM 
does  not  have  to  specifically  address  the  projects  in 
relation  to  the  types  and  extent  of  contamination  present, 
site  characteristics,  and  the  expected  future  use  of  the 
site.  Likewise,  the  uncertainties  associated  with  the 
restoration  method  chosen,  the  risk  assessment,  feasibility, 
cost,  and  other  criteria  of  interest  were  not  specifically 
addressed  in  either  model.  It  was  assumed  that  the 
proponents  of  the  project  submitted  an  appropriate  'expected 
value'  for  these  aspects  of  the  project.  Linking  the  MCDM 
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to  probability  density  functions  would  likely  not  improve 
the  resulting  priorities  at  this  stage  of  model  development 
and  could  overly  complicate  it  in  the  eyes  of  the  IRP 
community. 

MCDM  Procedures .  In  light  of  the  numerous  and 
conflicting  opinions  for  criteria  to  be  used  in  the 
resulting  MCDM,  two  procedures  were  chosen  for  this 
research.  Each  procedure  has  resulted  in  a  usable  model 
which  can  be  modified  to  fit  changing  future  requirements, 
new  information,  or  valid  criticism  from  stakeholders. 

The  strength  of  the  Delphi  procedure  lies  in  its 
ability  to  identify  criteria  for  the  model.  It  solicits  and 
incorporates  a  broad  range  of  ideas  from  DERP  experts  and 
builds  a  consensus  among  them  for  the  final  model.  In 
contrast,  the  strength  of  the  MAUT  procedure  was  its  ability 
to  quantify  the  preference  among  the  criteria  by  means  of 
pairwise  comparisons  and  to  check  the  independence  of  the 
variables . 

The  results  of  the  Delphi  are  limited  by  the  variance 
among  the  experts  for  the  utility  and  weights  to  be  applied 
to  the  criteria.  While  a  small  variance  indicates  general 
agreement  and  increases  the  confidence  of  the  measure, 
larger  variances  cast  doubt  as  to  whose  judgment  is 
representative  of  the  population.  In  addition  to  the 
variance  of  the  experts,  multiple  expert  interpretations  of 
the  best  solution  must  be  aggregated  into  a  consensus  single 
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group  interpretation.  Each  expert  uses  their  own  frame  of 
reference,  and  individual  judgments  of  the  best  solution  may 
not  be  reflected  in  the  aggregate.  (Lewis,  1993:1)  A 
limiting  factor  in  the  MAUT  procedure  stems  from  the  fact 
that  a  single  decision  maker's  opinion  is  solicited  for  the 
resulting  value  function.  There  is  no  check  on  the  decision 
maker's  inputs  or  the  analyst's  interpretation  of  the 
responses,  although  follow-on  interviews  were  used  in  an 
attempt  to  minimize  these  problems. 

MCDM  Results.  The  MCDM  weighting  factors  derived  in 
Chapter  4  are  as  follows.  The  utility  functions  shown  in 


the  text  are  not  repeated  here. 

Delphi  weighting  factors: 

1)  Threat  to  Human  Health  and  the  Environment  =  0.183 

2)  Expected  Reduction  of  Risk  =  0.164 

3)  Technical  Feasibility  =  0.155 

4) Cost  =  0.146 

5)  Federal  Facility  Agreement  =  0.142 

6)  Community  Acceptance  =  0.091 

7)  Mission  Impact  =  0.091 

8)  Socioeconomic/Political  Inpacts  =  0.027 


MAUT  weighting  factors: 

1)  Threat  to  Human  Health  and  the  Environment  =  1.0 

2)  Cost  =0.0 

3 )  MAJCOM  Priority  =0.0 
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Both  rank  discretionary  projects  on  the  criteria  of 
importance  to  the  experts  and  decision  makers.  However, 
many  projects  submitted  for  funding  do  not  need  to  be 
prioritized.  A  DSS  provides  for  these  non-discretionary 
type  decisions . 

A  final  DSS  was  not  developed  as  part  of  this  research. 
Instead,  the  research  focused  on  the  MCDM  portion  of  the 
DSS.  A  DSS  framework  was  under  development  at  the  Air  Force 
installation  restoration  program  (IRP)  headquarters.  The 
DSS  provides  funding  for  'must  pay'  requirements  such  as 
personnel,  program  management,  long-term  operations  and 
monitoring,  and  initial  site  investigation  work  (Owendoff, 
1993b) .  These  requirements  must  be  funded  to  provide 
continuity  in  the  IRP,  to  supply  the  information  needed  to 
prioritize  future  work,  and  to  fulfill  the  obligations 
incurred  under  earlier  work.  After  these  requirements  have 
been  funded,  the  DSS  attempts  to  prioritize  IRP  projects 
based  on  risk.  Using  an  MCDM  proposed  by  this  research 
could  improve  that  portion  of  the  DSS,  or  at  least  provide  a 
rationale  for  using  only  DPM  to  prioritize  the  projects. 

Using  the  proposed  DSS  in  conjunction  with  the  MCDM 
does  meet  the  objectives  of  this  research.  First  and 
foremost,  it  provides  a  distinct,  objective,  funding 
hierarchy.  Second,  the  projects'  priorities  will  be 
consistent  with  the  current  laws  and  regulations  governing 
the  program.  The  projects  submitted  will  have  been  approved 
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by  both  the  installation  and  Major  Command  (MAJCOM)  level 
managers  as  meeting  the  statutory  and  Department  of  Defense 
(DoD)  regulatory  requirements.  Furthermore,  projects 
submitted  for  high  risk  sites  will  likely  have  been  approved 
by  a  technical  review  committee  composed  of  Air  Force,  U.S. 
and  state  environmental  protection  agency  (EPA)  personnel, 
and  community  representatives.  Finally,  the  DSS  and 
resulting  prioritization  should  be  acceptable  to  the 
MAJCOMs ,  Congress,  the  EPAs,  and  the  public.  Many  of  these 
groups  participate  in  the  process  and  most  of  their  concerns 
have  been  addressed.  More  will  be  said  about  improving  this 
portion  of  the  DSS  in  the  recommendations  section. 

FgcQimendfltiQns  For.  Further  Study 

Criteria.  Further  research  could  include  an  attempt  to 
incorporate  project -specific  criteria  such  as  the 
suitability  of  the  proposed  work  given  the  site  and 
contaminant  characteristics.  Furthermore,  an  attempt  could 
be  made  to  model  the  uncertainties  involved  in  the  various 
project  attributes.  Finally,  new  metrics  could  be  examined 
for  measuring  the  criteria  of  interest. 

Delphi  Procedures .  The  principal  improvement  in  this 
area  of  the  research  would  entail  expanding  the  scope  of 
respondents  to  include  a  wider  spectrum  of  stakeholders. 

More  regulatory  personnel,  industry  consultants,  community 
representatives,  and  environmental  groups  should  be  included 


116 


to  better  represent  the  interests  of  the  stakeholders  and 
build  support  for  the  resulting  model.  Additionally,  the 
respondents  should  be  required  to  specify  their  choice  of 
metrics  for  the  various  criteria  proposed.  In  this  manner, 
several  related  criteria  and  other  subjective  criteria  may 
be  further  collapsed  before  an  MAUT  procedure  is  employed, 
and  the  weights  given  to  those  criteria  may  change. 

MAUT  Procedures .  Given  the  time  constraints  of  this 
research  and  the  level  of  experience  of  the  analysts,  the 
MAUT  procedure  suffered  from  inadequate  criteria  and  a 
single  decision  maker.  Improvements  would  include  following 
the  Delphi  procedure  with  the  MAUT  questionnaire  so  that 
resulting  Delphi  criteria  could  be  utilized  in  the  pairwise 
comparisons.  The  subjects  for  the  MAUT  questionnaire  should 
be  expanded  to  include  either  the  original  Delphi 
respondents  or  someone  at  the  next  higher  level  of 
management  within  their  organization.  Examining  the  value 
functions  of  additional  subjects  should  increase  the 
generalizability  of  the  results  and  reduce  the  tendency  for 
the  function  to  change  as  new  decision  makers  take  office. 
The  techniques  for  this  type  of  interactive,  multi-person, 
multiobjective  decision  research  have  been  advanced  in 
several  recent  studies  (Lewis  and  Butler,  1993:1). 

Construction  of  DSS.  Because  the  DERP  involves  non¬ 
discretionary  as  well  as  discretionary  projects,  the  use  of 


an  MCDM  for  prioritization  is  limited  to  a  subset  of  the 
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overall  range  of  projects  submitted.  Therefore,  a  DSS  that 
simplifies  use  of  the  MCDM  is  desirable.  This  research  left 
the  development  of  the  DSS  to  the  DoD  and  Air  Force  decision 
makers.  Future  research  could  entail  development  of  a  DSS 
in  addition  to  improvements  in  the  MCDM. 

CQP.ClUS.i.QP 

Prioritizing  IRP  projects  is  a  complex  and  arduous 
task.  While  it  is  not  difficult  to  identify  the  key 
objectives  of  the  program,  it  is  very  difficult  to  gain  a 
consensus  among  the  stakeholders  concerning  the  relative 
importance  of  those  objectives  and  the  metrics  to  be  used  in 
prioritizing  the  proposed  projects. 

This  research  has  utilized  two  widely  accepted  research 
techniques  in  developing  an  MCDM  to  resolve  this  problem. 

The  findings  indicate  two  distinct  solutions  that  could  be 
employed  at  the  Air  Force  decision  making  level  to  provide  a 
tentative  ranking  of  discretionary  projects.  Future 
research,  feedback,  and  modification  of  the  proposed  MCDMs 
will  give  the  Air  Force  and  DoD  a  widely  accepted,  objective 
means  of  ensuring  the  most  critical  DERP  projects  are  funded 
despite  shrinking  budgets. 
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This  appendix  contains  an  alphabetical  listing  of  the 
acronyms  used  in  the  text  and  an  explanation  of  their 
meaning . 
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AFC  EE 

AFIRM 

AFIT 

AFMC 

AHP 

ARAR 

ATSDR 

CERCLA 

CEVR 

CWA 

DERA 

DERP 

DOE 

DoD 

DPM 

DSMOA 

DSS 

EPA 

FFA 

FFER 

HRS 

IAG 

IRA 

IRP 

MAD 

MAJCOM 


Air  Force  Center  for  Environmental  Excellence. 

Air  Force  Installation  Restoration  Management. 

The  Air  Force  Institute  of  Technology. 

Air  Force  Materiel  Command. 

Analytic  Hierarchy  Process. 

Applicable  or  Relevant  and  Appropriate 
Requirement . 

Agency  for  Toxic  Substances  and  Disease  Registry. 

Comprehensive  Environmental  Response,  Compensation, 
and  Liability  Act  of  1980,  as  amended  (also  known  as 
' Superfund 1  Legislation). 

Civil  Engineering,  Environmental  Restoration  Branch. 
Clean  Water  Act. 

Defense  Environmental  Restoration  Account. 

Defense  Environmental  Restoration  Program. 

The  Department  of  Energy. 

The  Department  of  Defense. 

Defense  Priority  Model. 

Defense-State  Memorandum  of  Agreement. 

Decision  Support  System. 

The  Environmental  Protection  Agency. 

Federal  Facility  Agreement. 

Federal  Facilities  Environmental  Restoration. 

Hazard  Ranking  System. 

Interagency  Agreement. 

Interim  Remedial  Action. 

Installation  Restoration  Program. 

Median  Absolute  Deviation  About  the  Median. 

Major  Command. 
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MAUT 

MCDM 

NCP 

NPL 

OSWER 

PA/ SI 

PRP 

RAAS 

RACER 

RCRA 

RD/RA 

RI/FS 

RMS 

SARA 

SAB 

SAW 

SSAB 

TOPSIS 

use 


Multi-Attribute  Utility  Theory. 

Multicriteria  Decision  Model. 

National  Contingency  Plan. 

National  Priorities  List. 

Office  of  Solid  Waste  and  Emergewncy  Response. 

Preliminary  Assessment/Site  Inspection. 

Primarily  Responsible  Party. 

Remedial  Action  Assessment  System. 

Remedial  Action  Cost  Engineering  and  Requirements 
System. 

Resource  Conservation  and  Recovery  Act  of  1980,  as 
amended . 

Remedial  Design/Remedial  Action. 

Remedial  Investigation/Feasibility  Study. 

Root  Mean  Square 

Superfund  Amendments  and  Reauthorization  Act  of 
1986. 

Scientific  Advisory  Board. 

Simple  Additive  Weighting. 

Site-Specific  Advisory  Board. 

Technique  for  Order  Preference  by  Similarity  to 
Ideal  Solution. 

United  States  Code. 
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Appendix  B:  AFCEE  Remediation  Matrix  -  Hierarchy  of 
Preferred  Alternatives 


This  appendix  contains  a  copy  of  the  AFCEE  remediation 
matrix  showing  the  preferred  remediation  alternatives  for 
specific  contamination  scenarios. 
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PREFERRED  ALTERNATIVES 


Appendix  C. _ Bre-Questionnaire  for  MAUT  Procedure 


This  appendix  contains  the  pre-questionnaire  used  in  the 
MAUT  procedure  to  familiarize  the  decision  maker  with  the  MAUT 
process,  and  to  elicit  the  criteria  of  importance  in  the  DERP. 

The  responses  indicated  in  bold,  italicized  type  are  primarily 
attributable  to  the  decision  maker.  Note:  not  all  questions  were 
asked/answered . 
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Prequestionnaire  Outline: 


Goals; 

1.  Familiarize  Col.  Owendoff  with  concepts  and  techniques. 

-  x,  y,  and  z  space  (alternatives,  criteria,  preference) 

-  ordinal/cardinal,  discrete/continuous,  independence  among 
variables  (preferential,  utility) 

2.  Determine  objectives,  criteria,  and  metrics. 

3.  Discuss  potential  DSS  framework. 


Questions  for  Col.  Owendoff: 


1)  What  is  the  alternative  space,  or  in  other  words,  do  you 
prefer  scoring  sites  or  projects?  Prefer  to  prioritize  by 
aite. 


2)  What  is  the  criterion  set,  or  in  other  words,  what  parameters 
do  you  consider  important  enough  to  include  in  a  decision  support 
system?  Which  must  be  met  at  a  minimum  level  and  which  can  be 
evaluated  on  a  graduated  scale?  What  measures  should  we  use  to 
evaluate  them?  (want  to  reduce  #  of  judgements  required  and 
complexity  of  DSS) 


a)  Protection  of  human  health  and  the  environment.  (DPM 
score  at  site?  DPM  before  and  after?  Urgency  measure?) 

important,  DPM  &  urgency. 

b)  Meeting  ARARs .  (Peer  review?  Base  judgement?) 

c)  Long-term  effectiveness/permanence,  (post-project  DPM?) 

Effectiveness  is  important,  but  ?  metric. 

d)  Reduction  in  toxicity,  mobility,  volume  of  waste, 
(contaminant  remaining?  post -project  DPM?) 


e)  Short-term  effectiveness/risk  of  action.  (Peer  review?) 


f)  Implementability/ feasibility .  (Peer  review?,  MAJCOM 
review?  Past  track  record?) 

Peer  review  already  used  for  RA's. 

g)  Cost-effectiveness.  ($'s  for  project?,  $'s  for  overall 

fix? ) 

Important,  $  for  project. 

h)  Acceptability  by  State.  (IAW  agreements?) 

Maybe,  but  agreements  weren't  built  with  today's 
fiscal  constraints  in  mind. 

i)  Acceptability  by  public.  (IAW  CERCLA/ROD?,  MAJCOM  review? 
Urgency  measure?) 
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5)  Are  the  parameters  of  importance  discrete  or  continuous? 

(has  been  funding  as  continuous  but  may  be  forcing  the  MAJCOMs  to 
make  “discrete"  choices) 

6)  What  actions  are  underway  to  develop  priority  model  for  AF? 

Working  with  ACC  to  use  DPM  and  DPM  Quick  to 
prioritise  projects. 

Other  Comments  (interactive  discussion/brainstorming) : 

-  Keep  DSS  simple,  ensure  consistency  across  commands  (no 
gaming) 

-  Could  use  a  system  where  projects  are  prescreened  and  the  MCDM 
could  be  applied  to  those  that  aren't  "must-pay"  projects  or 
projects  that  could  obviously  be  deferred: 

--  Fund  management  &  monitoring  up-front. 

--  Rank  "best"  projects  IAW  MCDM.  Results  could  be 
categorized 

as  "superior"  projects  for  immediate  funding,  "inferior" 
projects  for  deferral,  and  "marginal"  projects  for 

further 

consideration.  MCDM  would  provide  breakout  for  all  three 

and 

initial  ranking  for  "marginal"  projects. 

-  Research  should  develop  solutions  that  we  can  evaluate  and 
get  ideas  from. 
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Appendix  D.  Questionnaire  for  MAUT  Procedure 


This  appendix  contains  background  information  and  the 
questionnaire  used  to  elicit  the  preference  structure  of  the 
decision  maker  for  use  in  the  MAUT  procedure. 

The  decision  maker's  responses  are  indicated  by  bold,  italicized, 
and  underscored  type. 
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The  criteria  used  to  evaluate  projects  submitted  for 


DERA  funding  were  determined  from  the  literature  review  and 
reflect  an  aggregation  of  the  broad  range  of  suggested  criteria. 
By  combining  similar  criteria,  it  should  be  possible  to  maintain 
independence  among  the  criteria  and  simplify  the  model.  The 
criteria  have  been  left  general  enough  to  apply  to  all  types  of 
projects  while  still  providing  a  discriminatory  ability.  The 
criteria  chosen  include: 

a.  Site  risk.  This  criterion  has  been  identified  in  most 
previous  studies,  is  a  statutory  consideration,  and  is  emphasized 
in  Air  Force  regulations  and  guidance.  A  project  associated  with 
a  high  risk  site  will  be  assumed  to  be  strongly  correlated  with 
protection  of  human  health  and  the  environment,  risk  reduction, 
and  reduced  future  liability. 

b.  Major  Command  (MAJCOM)  priority.  This  measure  is  an 
attempt  to  combine  several  subjective  criteria  mentioned  in  the 
literature  while  recognizing  the  difficulty  of  establishing  a 
meaningful,  quantitative  measure  to  those  criteria.  MAJCOMs  with 
50  or  more  sites  would  be  asked  to  rank  the  projects  submitted 
each  fiscal  year  on  the  basis  of  the: 

1)  urgency  of  the  requirement  (whether  or  not  it  is  required 
in  the  current  fiscal  year  by  statute,  agreement,  or  as  a  result 
of  other  pending  actions  like  new  construction  on  the  site) , 

2)  feasibility  of  the  technology  to  be  employed, 

3)  political  sensitivity  of  the  site, 

4)  compliance  with  established  planning  documents  and 
concurrence  of  the  technical  review  committee  and  affected 
public,  and 


5)  environmental  equity  (site  goals  are  commensurate  with 
other  sites), 

6)  overall  compliance  with  Air  Force  goals  (10%  of  the  sites 
closed  each  year,  60%  of  the  funds  spent  on  remedial  actions,  and 
complete  all  sites  by  the  year  2000) . 

MAJCOMs  and  other  activities  with  fewer  than  50  sites  would  have 
their  projects  evaluated  and  prioritized  by  a  committee  from  the 
Air  Force  Center  for  Environmental  Excellence  ( AFCEE ) .  AFCEE  has 
a  good  understanding  of  overall  Air  Force  requirements  and 
provides  technical  support  to  many  of  the  smaller  commands. 

c.  Cost.  The  need  for  a  prioritization  model  is  predicated 
on  an  overall  funding  shortfall.  The  literature  consistently 
cited  cost  effectiveness  as  an  important  factor  to  the  resulting 
decision,  and  several  studies  used  cost/benefit  ratios  to 
prioritize  the  alternatives.  This  model  will  not  seek  to 
optimize  the  ranking  of  projects  by  using  costs  in  this  manner; 
rather,  the  decision  maker's  utility  for  project  costs  will  be 
combined  with  the  other  factors  in  a  determination  of  the  overall 
value  function. 

Metrics :  The  following  metrics  were  used  to  represent  the 
three  criteria  selected  for  use  in  the  decision  support  system. 
The  metrics  were  selected  because  they  can  be  represented  on  a 
ratio  scale.  The  appropriate  scaling  functions  and  weights  will 
be  determined  from  the  questionaire . 

a.  Risk;  Risk  will  be  measured  using  a  DPM  score.  DPM  was 
developed  to  assist  DoD  in  prioritizing  sites  and  is  a  reflection 
of  the  overall  risk  presented  by  the  site.  While  a  reduction  in 
risk  is  perhaps  a  better  metric  for  projects  like  removal  actions 
and  remedial  actions,  other  categories  of  projects  have  no 
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potential  to  reduce  risk.  Therefore,  the  site  risk  represented 
by  the  DPM  score  will  be  used  for  all  projects. 

b.  MAJCOM  priority:  The  MAJCOM  priority  will  be  converted 
into  a  value  between  zero  and  one  with  the  following  equation: 

Value  =  1  -  [priority  /  total  number  of  projects] 

A  MAJCOMs  highest  priority  project  will  have  a  value  almost  equal 
to  one.  The  lowest  priority  project  will  have  a  value  of  zero. 
Since  this  process  establishes  a  zero  to  one  scale  and 
establishes  mid-management's  utility  for  the  projects,  the 
questionaire  will  only  be  used  to  determine  the  decision  maker's 
preference  for  the  weight  to  given  to  this  criterion  in  the 
overall  utility  function. 

c.  Cost:  Cost  is  the  most  straightforward  measure,  yet  the 
magnitude  of  the  potential  disparity  between  the  highest  cost 
project  and  the  lowest  could  reduce  the  discriminatory  power  for 
low  and  moderate  cost  projects  (e.g.  if  the  most  expensive 
project  cost  $10  million  and  the  scale  was  based  on  total  cost 
divided  by  the  most  expensive  project,  all  projects  under  $1 
million  would  be  be  scaled  between  0.9  and  1.0).  Therefore,  in 
addition  to  weight,  a  utility  scale  associated  with  costs  must  be 
determined  from  the  questionnaire  and  it  may  be  non-linear. 
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Questionnaire  to  Determine  A  Value  Function 
For  IRP  Prioritization 

A.  Assume  for  the  following  questions  you  are  comparing  two 
projects  that  fall  very  close  to  the  funding  cutoff  point. 

1.  Site  A  has  a  DPM  score  of  35.  Site  B  under  a  different 
MAJCOM  has  a  DPM  score  of  15.  Similar  RD  projects  are  submitted 
for  both  sites  at  a  cost  of  $100,000.  What  difference  in  MAJCOM 
priorities  would  make  you  chose  the  project  at  the  Site  B  over 
Site  A? 

would  fllaaze.  shoes.  Slte-  A, 

b.  A  difference  of  at  least  _ /100. 

c.  I  would  be  indifferent  to  the  two  projects. 

2.  Given  the  same  DPM  scores  as  in  question  1  but  RA  project 
costs  of  $2.5  million,  what  difference  in  MAJCOM  priority  would 
make  you  chose  the  Site  B  over  Site  A? 

a.  I  would  always  chose  Site  A. 

b.  A  difference  of  at  least  _ /100. 

c.  I  would  be  indifferent  to  the  two  projects. 

3.  Site  A  has  a  DPM  score  of  15.  A  similar  Site  B  has  the  same 
DPM  score.  The  RI/FS  project  submitted  for  Site  A  cost  $150,000 
and  the  RA  project  for  Site  B  costs  $2,000,000.  What  difference 
in  MAJCOM  priorities  would  make  you  chose  the  project  at  Site  B 
o  'r  Site  A? 

a.  J  would  always  chose  site  A. 

b.  A  difference  of  at  least  _ /100. 

c.  I  would  be  indifferent  to  the  two  projects. 

4.  Site  A  has  a  DPM  score  of  35.  A  similar  Site  B  has  the  same 
DPM  score.  The  RI/FS  project  submitted  for  Site  A  cost  $150,000 
and  the  RA  project  for  Site  B  costs  $2,000,000.  What  difference 
in  MAJCOM  priorities  would  make  you  chose  the  project  at  Site  B 
over  Site  A? 

a.  I  would  always  chose  Site  A. 

b.  A  difference  of  at  least  _ /100. 

c.  I  would  be  indifferent  to  the  two  projects. 

5.  Site  A  has  a  DPM  score  of  30.  Site  B  under  a  different  MAJCOM 
has  a  DPM  score  of  15.  The  same  priority  of  50  out  of  100  was 
assigned  to  the  projects  by  their  respective  MAJCOMs.  How  much 
would  you  have  to  save  to  make  you  chose  the  RI/FS  project  at  the 
Site  B  over  the  $500,000  RI/FS  project  at  Site  A? 

a.  I  would  always  choee  Site  A. 

b.  A  difference  of  at  least  $ _ . 

c.  I  would  be  indifferent  to  the  two  projects. 
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6.  Given  the  same  DPM  scores  as  in  question  5  but  priorities  of 
20  out  of  100,  what  difference  in  costs  would  make  you  chose  the 
Site  B  over  the  $500,000  RI/FS  at  Site  A? 

a.  I  would  always  chose  Site  A. 

b.  A  difference  of  at  least  $ _ . 

c.  X  would  be  indifferent  to  the  two  projects. 

No  difference  from  question  5.  No  utility  In  MAJCCH  priority. 

B.  The  following  questions  will  be  used  to  determine  how  much 
the  value  you  associate  with  each  criterion  changes  as  the  other 
criteria  change.  Assume  for  the  following  questions  you  are 
concerned  with  ranking  two  projects  along  just  one  criterion. 
Determine  the  degree  you  would  rank  the  projects  differently 
based  on  only  the  subject  criterion  while  noting  the  change  in 
the  other  criteria. 


7.  MAJCOM  X  submits  Project  A  for  an  RA  at  $4,000,000.  MAJCOM  Y 
submits  Project  B  for  an  RA  at  $7,000,000. 


Case 

PPM- 

Score 

Priority 

A 

B 

A 

B 

#1) 

30 

40 

5 

10 

#2) 

20 

20 

20 

40 

#3) 

50 

10 

1 

20 

8 .  MAJCOM 

X  submits 

Proji 

Utility  Associated  With 
The  Difference  in  Project  Cost 
None  -  Some  -  Great 


X- 

X- 

X 


-X 

-X- 

-X- 


-x- 

-X- 

-X- 


-X 

-X 

-X- 


-X 

X 

-X 


MAJCOM  Y  submits  Project  B  as  their  middle  priority  project  (50 
out  of  100) . 

Utility  Associated  With 

case  ppm  Score  cost  The. Difference  in  MAJCOM  Priority 

A  B  A  B  None  -  Some  -  Great 


#1)  30  10  $3M  $.2M  X - X - X - X - X 

#2)  20  20  $2M  $1M  X - X - X - X - X 

#3)  50  10  $5M  $.2M  X - X - X - X - X 


9.  MAJCOM  X  submits  Project  A  with  a  site  DPM  score  of  40. 
MAJCOM  Y  submits  Project  B  with  a  site  DPM  score  of  20. 


Case 

Pri.c.r.itY 

Cost 

A 

B 

A 

B 

#1) 

30 

10 

$3M 

$.5M 

#2) 

20 

20 

$2M 

$1M 

#3) 

50 

80 

$ .  5M 

$.2M 

Utility  Associated  With 
The  Difference  in  Site  DPM  Scores 
None  -  Some  -  Great 

X - x - X - X - X 

X - x - X - X - X 

X - x - X - X - X 
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C.  Assume  for  the  following  questions  that  you  are  faced  with  a 
situation  where  you  have  a  choice  between  a  project  with  an 
uncertain  outcome  and  a  project  with  an  outcome  that  is 
guaranteed.  You  are  to  chose  the  cost  or  DPM  score  that  you 
would  associate  with  the  guaranteed  alternative  that  would  make 
you  indifferent  between  it  and  the  uncertain  alternative. 

10.  You  have  a  choice  of  contracting  options  on  a  “turn-key“ 
project  at  a  newly  discover  site.  You  can  chose  a  “cost-plus" 
type  option  that  places  all  the  risk  on  the  government  for  the 
cost  incurred  at  the  site.  If  it  turns  out  the  initial  report 
was  wrong,  your  cost  is  nothing.  If  it  turns  out  that  you  have 
significant  contamination,  your  cost  will  be  $10  million  to  clean 
it  up.  There's  a  50/50  chance  for  either  situation.  You  also 
have  the  option  of  a  “ f irm-f ixed-price“  contract  whereby  the 
contractor  would  assume  all  the  risk  for  the  site.  You  would  pay 
him  a  set  fee.  The  contractor's  profit  equals  the  fee  your 
willing  to  pay  minus  his  costs  for  cleaning  up  any  contamination 
found.  If  the  site  is  clean,  his  profit  equals  the  fee  you've 
chosen.  If  it's  contaminated,  he  will  lose  the  difference 
between  the  fee  your  willing  to  spend  and  the  $10  million.  How 
much  would  you  be  willing  to  pay  him  for  the  “firm-fixed-price" 
contract? 

$5Jtf _  vs  50/50  chance  for  $0  or  $10M 

Now  suppose  you  know  that  the  cost  would  not  exceed  $5  million  to 
clean  up  the  site.  How  much  would  you  be  willing  to  spend  for 
the  guaranteed  outcome  (FFP  contract)? 

$3.5Jf  vs  50/50  chance  for  $0  or  $5M 
What  if  the  cost  would  not  exceed  $2.5  million? 

S 1 . 25M  vs  50/50  chance  for  $0  or  $2.5M 

What  if  the  cost  would  not  exceed  $1  million? 

$  .5Bf _  vs  50/50  chance  for  $0  or  $1M 

New  information  is  uncovered  that  shows  that  the  contamination 
will  not  be  a  problem.  However,  you  still  need  to  document  that 
to  the  regulators  before  you  close  the  site  out.  The  cost  of 
documentation  could  either  $0  or  $500,000  with  a  50/50  likelihood 
for  either.  How  much  would  you  be  willing  to  pay  for  a  FFP 
contract  that  guarantees  the  reports  will  be  accepted? 

$,25K _  _  vs  50/50  chance  for  $0  or  $.5M 


What  if  the  cost  would  not  exceed  $250,000? 

$ . 125M  vs  50/50  chance  for  $0  or  $.25M 
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Now  suppose  that  you  authorized  the  contractor  to  initiate  work 
under  a  "cost-plus"  basis.  At  varying  stages  of  the  contract  you 
have  the  option  to  stop  work  and  change  the  contract  to  a  ■ firm- 
fixed-price"  contract.  How  much  would  you  be  willing  to  pay  for 
the  FFP  contract  under  the  following  scenarios: 


50/50 

chance 

costs 

would 

be 

either 

$ .  25M 

or  $ . 5M? 

$*375* 

50/50 

chance 

costs 

would 

be 

either 

$ . 5M  or  $1M? 

$  .  75M 

50/50 

chance 

costs 

would 

be 

either 

$1M  or 

$2 . 5M? 

$1, 75* 

50/50 

chance 

costs 

would 

be 

either 

$2 . 5M 

or  $5M? 

$3,75* 

50/50 

chance 

costs 

would 

be 

either 

$5M  or 

$10M? 

$1*5* 

11.  An  IRP  contractor  has  recently  made  headlines  with  a  new 
strain  of  microbes  that  he  claims  will  "render  any  contamination 
at  any  site  harmless."  You're  skeptical,  but  you're  considering 
giving  him  a  try.  You  have  a  contaminated  site  that  has  a  DPM 
score  of  50.  You  believe  that  the  new  contractor  has  a  50/50 
chance  of  success  at  the  site.  If  he  succeeds,  the  site  can  be 
closed  out  permanently.  If  not,  you've  neither  lost  or  gained 
anything  at  the  site.  Another  contractor,  one  that  you've  dealt 
with  before,  also  offers  to  work  at  the  site  for  you  under  the 
same  terms  as  the  new  contractor.  You  know  this  contractor 
cannot  completely  clean  up  the  site  with  his  technology,  but  he 
can  reduce  the  DPM  score  significantly.  At  what  reduced  DPM 
score  would  you  be  indifferent  to  the  two  proposals? 

DPM  of  30  vs  50/50  chance  of  0  or  50 

Before  you  made  your  decision  in  the  above  problem,  new 
information  came  in  that  reduced  the  existing  DPM  score  at  the 
site  to  25.  The  situation  and  odds  remain  as  before.  At  what 
reduced  DPM  score  would  you  be  indifferent  to  the  two  proposals? 

DPM  of  15  vs  50/50  chance  of  0  or  25 

Once  more,  new  information  surfaced.  Now  the  DPM  is  15.  At  what 
reduced  DPM  score  would  you  be  indifferent  to  the  two  proposals? 

DPM  of  10  vs  50/50  chance  of  0  or  15 

The  new  contractor  with  the  "miracle  microbes"  has  visited  your 
site  and  has  revised  his  claims.  You  now  have  a  choice  of  using 
the  new  contractor  who  says  he  can  reduce  the  DPM  score  f  z  ">m  50 
to  25  or  the  old  contractor  who  can  guarantee  his  work.  You 
still  feel  the  new  contractor  has  a  50/50  chance  of  success.  At 
what  reduced  DPM  score  would  you  be  indifferent  to  the  two 
proposals? 


DPM  of  40  vs  50/50  chance  of  25  or  50 
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You’ve  been  promoted  to  MAJCOM  and  have  many  sites  to  consider. 
The  scenario  is  the  same  at  each.  Please  indicate  the  reduced 
DPM  you’d  be  indifferent  to  given  the  new  contractors  claims  with 
their  50/50  uncertainty: 


DPM 

of 

20 

vs 

50/50 

chance 

of 

15 

or 

25 

DPM 

of 

30 

vs 

50/50 

chance 

of 

25 

or 

35 

DPM 

of 

J5 _ 

_  VS 

50/50 

chance 

of 

35 

or 

50 

D.  As  in  Section  A.,  again  assume  for  the  following  questions 
that  you  are  comparing  two  projects  that  fall  very  close  to  the 
funding  cutoff  line. 


12.  An  RA  project  for  Site  A  with  a  DPM  score  of  30  has  been 
submitted  at  a  cost  of  $4,000,000.  An  RI/FS  project  for  Site  B 
with  a  similar  DPM  score  of  30  has  been  submitted  at  a  cost  of 
$1,500,000.  What  MAJCOM  priorities  would  make  you  indifferent 
between  the  two  projects? 

NONE  Site  A:  _  vs  Site  B:  _ 

If  the  cost  for  Site  A  were  only  $2,000,000? 

NONE  Site  A:  _  vs  Site  B:  _ 


13.  An  RD  project  for  Site  A  with  a  DEW  score  of  15  has  been 
submitted  at  a  cost  of  $1,000,000.  An  RI/FS  project  for  Site  B 
with  a  DPM  score  of  35  has  also  been  submitted  at  a  cost  of 
$1,000,000.  What  MAJCOM  priorities  would  make  you  indifferent 
between  the  two  projects? 

HQNB  Site  A:  _  VS  site  B:  _ 

If  the  DPM  at  Site  A  was  25? 

NONE  site  A:  _  VS  Site  B:  _ 


14.  An  RD  project  for  Site  A  with  a  DPM  score  of  15  has  been 
submitted  with  a  priority  of  50  out  of  100.  An  RI/FS  project  for 
Site  B  with  a  DPM  score  of  35  has  also  been  submitted  with  a 
priority  of  10  out  of  100.  What  project  costs  would  make  you 
indifferent  between  the  two  projects? 

NONE  Site  A:  _  vs  Site  B:  _ 

If  the  DPM  at  Site  A  was  25? 

NONE  Site  A:  _  vs  Site  B;  _ 
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E.  As  a  control  on  the  calculations  of  your  utility  function, 
please  take  a  few  minutes  to  answer  the  following  questions: 


15.  Which  of  the  criteria  do  you  feel  is  most  important  in 
ranking  IRP  projects? 

DPM  Score  at  a  Site  X  MAJCOM  Priority  _  Cost  _ 


The  least  important? 

DPM  Score  at  a  Site  _  MAJCOM  Priority  _  Cost  _J[ 

16.  Do  you  believe  there  are  other  more  important  criteria  that 
aren't  accounted  for  in  the  questionaire?  (Please  list  them  in 
order  of  importance.) 

All  of  the  important  criteria  hare  been  used  (risk, cost) .  Need 
to  define  the  statistical  ranges  for  DPM  values  (high,  med,  low)  . 


17.  How  much  flexibility  would  you  estimate  a  project  manager 
has  in  scaling  back  their  projects  if  a  fair  share  allocation  of 
all  eligible  projects  is  used  rather  than  a  ranking  method? 

Very  Flexible. 


Proposed  alternative  priority  system: 


manpower 

mgrfc  (MSP,  TRC,  CRP) 

15 5f 


¥X7SI  (SARA  Docket) 

RI/FS,  RD/rA.  where  DPM>40 
TPS 


Off-base  migration 

Source  remova 1/free  product  IRA s 

RI  presumptive  remedies 

AT  — >  RA  PPM  between ~ 10  and  40,  In  situ  remedies 
RI  — >  RA  DPM  between  10  and  40,  iton-In  situ  remedies 
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Appendix  E. _ Suggested  List  of  DERP  Experts 


This  appendix  contains  the  electronic  mail  note  from 
Captain  Rob  Wilson  of  AFIT/DE  with  the  list  of  suggested 
DERP  experts  for  use  in  the  Delphi  procedure. 
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From :  SMTP% ■ rdwi 1 son@AFITDE . AFIT . AF . MIL “ 

To:  TELLINGS@CSC.AFIT.AF.MIL  <TELLINGS@CSC.AFIT.AF.MIL>, 

CC: 

Subj :  DERP  Experts 

Date:  05  Mar  93  11:25:28  EST 

From:  Capt  Rob  Wilson  < rdwi lson@AFITDE .AFIT. AF .MIL>, 

Subject:  DERP  Experts 

To:  TELLINGS@CSC.AFIT.AF.MIL 

<TELLINGS@CSC . AFIT .AF.MIL>, 

X-Office_CC:  Capt  Rob  Wilson 


Maj  Stuart  Nelson  HQ  USAF/CE  (202)767-4616  DSN-297-4616 

-6245  -6245 

Mr  Bob  James  OO-ALC/EMR  (801)777-6741  DSN  458-6741 

Mr  Mario  Ierardi  SM-ALC/EMR  (916)643-0531  DSN  633-0531 

Mr  Ron  Lester  645  ABW/EMR  (513)257-2201  DSN  787-2201 

Mr  Tony  Sculimbrene  645  ABW/EM  (513)257-5586  DSN  797-5627 

Mr  Ron  Sharpe  AFCEE/ESRR  (210)536-5218  DSN  240-5218 

Mr  Tony  Zugay  AFCEE/ESRR  (210)536-5211  DSN  240-5211 

Maj  Ross  Miller  AFCEE/ESRCT  (210)536-4329  DSN  240-4333 

Mr  Michael  Trimeloni  HQ  AFMC/CEVR  (513)257-7053  DSN  787-7053 

Also  recommend  MAJCOM  Reps. 

(MAJCOM  Rep  names  from  Maj  Stu  Nelson) 

MAJCOM  Reps: 

Ms  Martha  Shelby  USAFA/DEI  (719)472-4483  DSN  259-4483 

Mr  James  A  Rumbley  HQ  AU/CEV  (205)953-6976  DSN  493-6976 

Mr  Robert  Akridge  HQ  AFRES/CEV  (912)327-1070  DSN  497-1070 

Mr  Timothy  Corbett  HQ  AMC/CEVR  (618) 256-XXXX  DSN  576-XXXX 

Mr  Bob  Matsumoto  HQ  PACAF/CEVR  (808)486-8920  DSN  448-0470 

Mr  Bob  Moore  HQ  ACC/CEVR  (804)764-3108  DSN  574-3108 

Capt  Frank  Miles  HQ  ATC/CEV  (210)652-3240  DSN  487-3240 

Capt  William  A.  Thackerll00CES/CEV(202) 767-5443  DSN  297-5443 
Mr  K.  Jeff  Mundey  HQ  AFMC/CEVR  (513)257-7053  DSN  787-7053 
Mr  Gary  Hinkle  ANGRC  CEV  (301)981-8146  DSN  858-8146 

Maj  Mary  Vroman  HQ  SPACECOM/CEV  (719)554-5187  DSN  692-5187 


Note:  List  is  not  even  close  to  all  inclusive. 


Regards, 

Rob 
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Original  Memo 


To:  Capt  Rob  Wilson  From: 

TELLINGS@CSC . AFIT .AF.MIL 

Subject:  DERP  Experts  Date  Sent:  03/04/93 

RETURN-PATH :  <TELLINGS@CSC . AFIT . AF .MIL> 

RECEIVED:  FROM  CSC.AFIT.AF.MIL  BY  AFITDE.AFIT.AF.MIL  ;  04 
Mar  93  16:26:51  EST 

Date:  Thu,  4  Mar  1993  16:25:54  -0500  (EST) 

From:  TELLINGS@CSC.AFIT.AF.MIL 

Message-Id:  <930304162554.24a09bb8@CSC.AFIT.AF.MIL> 

Subject:  DERP  Experts 

To:  rdwilson@afitde.afit.af.mil 

X-Vmsmail-To:  SMTP%"rdwilson@af itde .af it .af .mil" 


Capt.  Wilson, 

Thanks  again  for  the  guidance  materials  on  IRP 
management .  I  was  hoping  I  could  get  you  to  repeat  that 
list  of  recommended  MAJCOM-level  DERP  experts,  and  any  other 
experts  you  think  might  be  helpful  in  reaching  a  consensus 
on  prioritization  criteria  for  DERP  projects. 

Also,  if  you  do  happen  to  get  a  copy  of  the  Air  Staff 
guidance  from  Maj .  Nelson,  I'd  sure  like  to  have  a  look  at 
it,  and  would  appreciate  it  if  you'd  give  a  call. 

Thanks , 

Tom  Ellingson  AFIT/ENV/GEE 
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Appendix  F, _ Cover  Letter  and  Questionnaire  for 

Delphi  Round  One 


This  appendix  contains  the  cover  letter  and 
questionnaire  used  to  determine  the  experts'  preferences  for 
criteria  to  be  used  in  the  prioritization  of  DERP  projects. 
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From:  Thomas  E.  Ellingson  (Commercial  5 1 3-427-004 1 ) 

AFIT/ENV-93S 

Wright-Patterson  AFB,  OH  45433 
Subj:  IRP  Prioritization  Research 
To. 

1 .  As  part  of  a  collaborative  master's  thesis,  my  colleague,  Mr  Gene  Gallogly,  and  1  are 
analyzing  the  decision  process  involved  in  prioritizing  Installation  Restoration  Program 
(IRP)  projects.  We  need  your  assistance  in  identifying  some  critical  factors  in  the 
prioritization  effort.  Please  take  a  few  minutes,  at  your  earliest  convenience,  to  read 
through  the  following  pages  and  answer  the  questions  included  with  this  package.  I  would 
appreciate  it  if  you  could  return  your  completed  questionnaire  via  fax  to  Tom  Ellingson, 
Box  4122,  AFIT/ENV  (Fax:  DSN  986-4943  /  Commercial  513-476-4943)  before 

28  May  1993. 

2.  As  I  am  sure  you  are  aware,  prioritizing  IRP  projects  is  a  rigorous,  time-consuming 
task.  Lack  of  critical  information,  the  diversity  of  projects,  the  multi-phased  nature  of 
projects,  and  the  uncertainties  inherent  in  the  various  technical  requirements  of  these 
projects  further  complicate  the  prioritization  effort.  Through  this  research,  my  colleague 
and  I  hope  to  enumerate  key  factors  which  would  assist  IRP  decision  makers  to  more 
easily  determine  the  relative  importance  of  a  given  project.  From  this  starting  point,  we 
hope  to  develop  a  decision  support  system  which  can  be  easily  manipulated  to 
accommodate  the  changing  requirements  of  the  Installation  Restoration  Program.  An 
executive  summary  of  the  findings  of  this  process  will  be  made  available  to  you,  if  desired, 
upon  completion  in  September  1993. 

3.  Iam  asking  for  your  assistance  in  participating  in  a  Delphi-style  process  which  will  take 
place  in  a  number  of  iterations. 

a.  This  first  round  of  questions  will  attempt  to  establish  a  set  of  criteria,  upon 
which  comparisons  between  IRP  projects  may  be  based  to  determine  priority.  You  are 
asked  to:  1)  examine  the  criteria  listed  and  add  others  which  you  feel  should  be  included, 
or  delete  those  criteria  which  you  feel  should  not  be  included,  when  considering  the 
priority  of  IRP  projects,  2)  suggest  a  means  of  measuring  each  criterion,  and  3)  assign  a 
weight  to  each  criterion  to  indicate  its  relative  importance.  Please  provide  a  short 
justification  for  any  criteria  which  are  added/deleted. 

b.  After  receiving  responses  from  the  first  round  of  questions,  I  will  compile  the 
responses  and  send  the  results  of  all  participants  back  to  you.  This  will  allow  you  to  see 
how  other  IRP  managers  placed  emphasis  on  the  various  criteria,  give  you  the  opportunity 
to  see  any  new  criteria  which  have  been  added,  and  also  allow  you  to  re-weight  the  new 
set  of  criteria  in  light  of  this  compiled  information. 
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c.  Subsequent  iterations  of  questions  may  be  used  to  further  refine  the  weighting 
factors,  or  to  gather  other  data  as  needed  for  the  construction  of  the  decision  support 
system. 

4.  Identifying  the  critical  factors  and  their  relative  importance  will  be  a  big  step  toward  a 
decision  support  system  to  assist  in  prioritizing  IRP  projects.  Awareness  of  these  factors 
will  also  help  define  the  information  requirements  necessary  to  support  the  decision 
making  process.  If  you  have  any  questions  or  concerns  about  this  research,  please  do  not 
hesitate  to  contact  my  thesis  advisor.  Dr.  Yupo  Chan,  at  DSN  785-2549  /  Commercial 
513-255-3362.  Thank  you  for  your  assistance. 


Thomas  E.  Ellingson  1  Atch 

AFIT  Graduate  Student  1.  Questionnaire  #1 
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Factors  in  Prioritizing  IRP  Projects 
Questionnaire  #1 


Name: 


Parti 


As  an  IRP  manager,  you  have  experienced  the  task  of  determining  the  relative 
importance  of  IRP  projects.  Please  examine  the  nine  criteria  listed  below  (promulgated  by 
the  National  Contingency  Plan  [The  NCP,  40  CFR  300.430(eX9)])  for  their 
appropriateness  in  prioritizing  Air  Force  IRP  projects.  Please  fed  free  to  delete  any 
criteria  which  you  fed  are  not  appropriate  for  any  reason,  or  to  add  any  criteria  which  you 
fed  have  been  excluded.  Please  provide  a  brief  justification  for  any  deletions  or  additions. 
Also,  please  suggest  a  means  of  measurement  for  all  criteria  which  you  fed  are 
appropriate  for  use  in  the  prioritization  of  IRP  projects. 

Examples.  Criterion:  Cost 

Measured  by  .  Total  lifetime  project  cost 

Criterion.  State  acceptance 

Measured  by:  Compliance  with  federal  facility  agreement  (yes/no) 


1.  Criterion:  Protection  of  human  health  and  the  environment 

Measured  by: 


2.  Criterion:  Compliance  with  applicable  or  relevant  and  appropriate 

requirements  (ARARs) 

Measured  by: 


3.  Criterion:  Long  term  effectiveness  and  permanence 

Measured  by: 
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* 


4.  Criterion: 
Measured  by: 


5.  Criterion: 
Measured  by: 


6.  Criterion: 
Measured  by  : 


7.  Criterion: 
Measured  by: 


8.  Criterion: 
Measured  by: 


9.  Criterion: 
Measured  by: 


Community  acceptance 


10.  Criterion: 
Measured  by: 
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Part  1  (cont.) 

11.  Criterion: 
Measured  by: 


12.  Criterion: 
Measured  by: 


13.  Criterion: 
Measured  by: 


14.  Criterion: 
Measured  by: 
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Part  II 


Using  your  answers  in  Part  I,  please  rank  the  factors  in  order  of  importance,  from 
most  to  least  important.  Weight  these  criteria  using  a  scale  of  I  to  100.  Assign  a  weight 
of  100  to  the  criterion  you  feel  is  most  important,  and  judge  all  others  in  light  of  that 
criterion.  For  example,  one  almost  as  important  might  be  9S;  half  as  important  would  be 
50. 


Rank  Ordered  Criteria:  Relative  weight: 

I  100 

2. 


3. 

4 

5. 


6. 


7. 

8 

9. 


10. 


11. 

12. 

13. 

14. 
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Appendix  G.  Cover  Letter  and  Questionnaire  for 
Delphi  Round  Two 


This  appendix  contains  the  cover  letter  and 
questionnaire  used  to  refine  the  criteria  and  weights  which 
were  determined  from  the  first  round  of  the  Delphi 
procedure . 
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From:  Thomas  E.  Eilingson  (Commercial  513-427-0041) 
AFIT/ENV-93S 

Wright -Patterson  AFB,  OH  45433 
Subj:  IRP  Prioritization  Research 


To: 


1.  Thank  you  for  your  response  to  the  first  Delphi  questionnaire.  I  have  compiled  the  results  from 
all  respondents,  and  based  upon  the  responses  gathered,  I  have  prepared  the  second  round  of 
questions.  Please  take  a  few  minutes  to  complete  the  attached  questionnaire  (Atch  1).  Please  note 
that  if  you  did  not  return  or  participate  in  the  first  questionnaire,  you  are  still  encouraged  to 
participate  in,  and  give  your  input  to  this  second  questionnaire.  I  would  appreciate  it  if  you  could 
return  your  completed  questionnaire  via  fax  to  Tom  Eilingson,  Box  4122,  AFIT/ENV  (Fax:  DSN 
986-4943  /  Commercial  5 13-476^943)  before  18  June  1993. 

2.  a.  As  can  be  seen  in  the  Round  One  Summary  of  Results  (Atch  2),  responses  to  the  Delphi 
questionnaire  produced  a  rich  list  of  criteria  considered  to  be  important  in  the  prioritization  of  IRP 
sites. 

b.  As  shown  in  the  Summary  of  Results,  there  were  eight  participants  in  Round  One.  These 
IRP  experts  identified  21  criteria  of  importance  in  IRP  site  prioritization  Twelve  of  these  criteria 
were  identified  by  more  than  one  respondent,  and  nine  of  tbs  criteria  were  identified  by  six  or  more 
respondents. 

c.  The  final  weight  assigned  to  each  criterion  is  simply  the  average  of  all  weights  assigned  to 
that  criterion  by  the  respondents. 

3.  In  compiling  the  resonses  to  the  first  round  of  questions,  I  noticed  that  a  lumber  of  criteria 
returned  by  different  responses  held  only  slight  semantic  differences.  As  the  coordinator  of  this 
Delphi  study,  I  attempted  to  narrow  die  breadth  of  the  study  by  'collapsing'  several  of  the  criteria 
into  similar  categories.  In  some  cases,  the  exact  wording  of  the  criteria  may  have  been  changed  in 
order  to  facilitate  categorization  of  the  criteria.  This  is  evident  in  Questionnaire  #2,  as  you  will 
notice  that  I  have  taken  the  21  criteria  from  Round  (tee,  and  collapsed  them  into  eight  criteria  for 
this  round.  The  eight  criteria  are  ranked  roughly  in  the  order  of  importance  as  derived  from  the 
criteria  returned  from  Round  One.  In  Attachment  3  I  have  given  a  definition  for  each  of  die 
criteria,  as  well  as  a  proposed  method  of  measurement,  arrived  at  by  taking  the  majority  of 
responses  for  that  particular  criterion. 

4.  For  this  round  you  are  given  the  information  from  all  Round  One  responses.  You  are  asked  to 
examine  these  data,  and  re-weight  the  the  IRP  prioritization  criteria  in  light  of  die  information 
compiled  from  the  first  round  of  questions.  If  you  do  not  agree  with  the  grouping  of  the  criteria  as 
they  now  stand,  the  definitions  presented,  or  the  proposed  means  of  measurement  for  die  criteria, 
please  feel  free  to  make  comments  on  Questionnaire  #2. 
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S.  If  you  have  any  questions  or  concerns  about  this  research,  please  do  not  ***»««**  to  contact  my 
thesis  advisor.  Dr.  Yupo  Chan,  at  DSN  785-2549  /  Commercial  513-255-3362.  Thank  you  for 
your  assistance. 


Thomas  E.  EUingson 
AFIT  Graduate  Student 


3  Atch 

1.  Questionnaire  #2 

2.  Round  One  Summary  of  Results 

3.  Criteria  Definitions  and  Proposed 
Means  of  Measurement 


G-3 


Attachment  1 


Development  of  Factor  Utility  in  Prioritizing  IRP  Projects 
Questionnaire  *3 


Name: _ _ 

The  following  is  a  summary  of  the  responses  to  the  recent  questionnaire  on  criteria  nvoived  in 
the  prioritization  of  Department  of  Defense  (DOD)  Installation  Restoration  Program  (IRP) 
projects.  The  list  of  criteria  has  been  'collapsed'  from  the  2 1  criteria  returned  in  response  to  Round 
One,  into  the  following  list  of  eight  criteria.  The  original  21  criteria  were  scored  haired  upon  the 
average  weight  assigned  by  the  respondents  (no  response  to  a  criterion  amounted  to  a  weight  of 
zero  being  assigned  by  that  respondent). 

Please  take  a  few  minutes  to  review  the  summary  (Atch  2),  and  weight  the  following  criteria 
using  a  scale  of  0  to  100.  Assign  a  weight  of  100  to  die  criterion  you  consider  to  be  the  most 
important  in  prioritizing  DOD  IRP  projects,  and  judge  all  others  according  to  that  •».  jidard  For 
example,  one  almost  as  important  might  be  given  a  weight  of  95;  one  half  as  important  would  be 
given  a  weight  of  50,  and  so  on.  Assign  a  weight  of  zero  to  any  criterion  that  you  fed  should  not 
be  considered.  Remember  that  the  order  shown  is  merely  an  approximation  of  the  composite 
ranking  from  the  first  round;  your  ordering  of  these  criteria  will  most  likely  not  match  the  printed 
order. 


Criteria  Relative  Weight 

1.  Threat  to  Human  Health  and  the  Environment  _ 

(Incorporates  Criteria  1,  10  &  12) 

2.  Technical  Feasibility  of  Site  Remediation  _ 

(Incorporates  Criteria  2  &  9) 

3.  Cost  _ 

(Incorporates  Criteria  2,  3, 4,  15  &  18) 

4.  Expected  Reduction  of  Risk  _ 

(Incorporates  Criteria  4, 6,  7  &  14) 

5.  Federal  Facility  Agreement  (FFA/lAG)  _ 

(Incorporates  Criteria  8, 11, 13,  16&17) 

6.  Community  Acceptance  _ 

(From  Criterion  5) 

7.  Socioeconomic/Political  Impacts  _ 

(Incorporates  Criteria  19  &  20) 

8.  Mission  Impact  _ 

(From  Criterion  21) 
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Attachment  3 


Definitions  of  Proposed  Criteria 
and 

Means  of  Measurement 


1.  Threat  to  Human  Health  and  the  Environment: 

The  relative  risk  presented  by  a  site  to  human  health  or  the  environment  Measured  by 
Defense  Priority  Model  (DPM)  score. 

2.  Technical  Feasibility  of  Site  Remediation: 

The  availability  of  an  acceptable,  proven  remediation  technology  which  will  meet  the  site- 
specific  applicable  or  relevant  and  appropriate  requirements  (ARARs).  Measured  in  binary 
(yes/no),  by  availability. 

3.  Cost: 

The  total  estimated  lifetime  cost  of  all  projects  associated  with  the  site.  This  includes  preyed 
costs  associated  with  implementing  and  maintaining  institutional  controls  (to  meet  ARARs),  such 
as  A/E  design  costs  and  remedial  action  costs;  costs  of  potential  delays  if  work  in  progress  is  not 
completed  in  a  timely  manner;  prevention  of  future  costs  through  immediate  action  or  pollution 
prevention;  etc.  Measured  by  present  value  of  the  total  dollar  cost  estimate  over  the  lifetime  of  the 
remediation  effort. 

4.  Expected  Reduction  of  Risk: 

The  estimated  amount  of  risk  (actual  or  potential)  which  can  be  reduced  through  remediation 
efforts.  This  will  include  both  short-  and  long-term  expected  effectiveness  of  remedial  actions. 
Measured  by  estimated  reduction  in  DPM  score. 

5.  Federal  Facility  Agreement  (FFA/IAG): 

The  presence  of  a  Federal  Facility  Agreement  (FFA)  or  an  Interagency  Agreement  (LAG) 
which  specifies  a  procedural  framework  and  time  schedule  for  the  site.  Measured  in  binary 
(yes/no),  by  presence. 

6.  Community  Acceptance: 

Degree  to  which  the  local  community,  including  all  stakeholders,  supports  the  proposed 
remediation  effort.  This  will  be  determined  by  communication  with  the  Technical  Review 
Committee  and  Public  Affairs.  Measured  as  high,  medium,  or  low  level  of  support. 

7.  Socioeconomic/Political  Impacts: 

The  anticipated  impacts  of  site  remediation  efforts  on  the  quality  of  life,  local  economy, 
and/or  local  politics  in  the  neighboring  areas.  Measured  as  varying  degrees  of  inpact  (positive,  no 
impact,  or  negative). 

8.  Mission  Impact: 

The  anticipated  impact  of  site  remediation  efforts  on  the  ability  of  various  base  organizations 
to  perform  their  designated  missions.  Measured  as  varying  degrees  of  impact  (positive,  no  impact 
or  negative). 
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Appendix  p, _ Cover  Letter  and  Questionnaire  for 

Delphi  Round  Three 


This  appendix  contains  the  cover  letter  and 
questionnaire  used  to  determine  the  experts'  utilities  for 
specific  data  points  within  the  criteria. 
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From:  Thomas  E.  Eliingson  (Commercial  513-427-0041) 
AFIT/ENV-93S 

Wright-Patterson  AFB,  OH  45433 
Subj:  1RP  Prioritization  Research 


1.  Thank  you  for  your  response  to  the  second  Delphi  questionnaire.  The  uniformity  of  the  results 
seem  to  indicate  a  great  degree  of  consensus  among  the  respondents,  and  based  upon  these 
responses,  I  have  prepared  the  third  and  final  round  of  questions.  Please  take  a  few  minutes  to 
complete  the  attached  questionnaire  (Atch  1).  Please  note  that  if  you  did  not  return  or  participate 
in  either  of  the  previous  questionnaires,  you  are  still  encouraged  to  participate  in,  and  give  your 
input  to  this  final  questionnaire.  I  would  appreciate  it  if  you  could  return  your  completed 
questionnaire  via  fax  to  Tom  Eliingson,  Box  4122,  AF1T/ENV  (Fax:  DSN  986-4943  / 
Commercial  513-476-4943)  before  02  July  1993. 

2.  a.  As  can  be  seen  in  the  Round  Two  Summary  of  Results  (Atch  2),  responses  to  the  second 
Delphi  questionnaire  seem  to  indicate  a  strong  consensus  regarding  the  criteria  considered  to  be 
important  in  the  prioritization  of  IRP  sites. 

b.  As  shown  ui  the  Summary  of  Results,  there  were  six  participants  in  Round  Two.  Each  of 
these  IRP  experts  confirmed  the  eight  criteria  of  importance  which  were  produced  by  collapsing  the 
results  of  the  previous  input  from  Round  (tee.  The  scoring  of  the  criteria  by  the  respondents 
indicates  that  there  is  some  degree  of  variance  of  opinion  among  the  experts,  but  taken  as  a  whole, 
the  rank  order  of  the  criteria  changed  very  little  between  Round  One  and  Round  Two.  This  seems 
to  indicate  a  strong  preference  structure  for  the  criteria  as  they  are  now  ordered. 

c.  Once  again,  the  final  weight  assigned  to  each  criterion  is  simply  the  average  of  all  weights 
assigned  to  that  criterion  by  the  respondents. 

3.  For  this  round,  you  are  given  the  information  from  all  Round  Two  responses  However,  no 
further  changes  will  be  made  to  the  criteria  of  importance  or  the  weighting  factors  for  these 
criteria.  Instead,  I  will  ask  you  to  examine  a  set  of  data  points  for  each  criterion  and  give  your 
utility  for  that  set  of  data  points.  The  utility  function  is  a  method  of  bringing  each  of  the 
measurements  for  the  criteria  to  a  common  scale,  and  is  essential  in  the  creation  of  a  model  for  use 
as  a  decision  support  system.  Further  directions  and  explanations  are  included  in  the  questionnaire 

4.  I  wish  to  give  my  deepest  thanks  for  your  participation  in  this  series  of  questionnaires.  Your 
input  to  the  research  being  done  by  myself  and  my  thesis  partner,  Mr  Gene  Gallogly,  has  been 
invaluable  to  our  research  effort,  and  will  hopefully  result  in  a  decision  support  system  that  will  be 
a  small  step  toward  better  Installation  Restoration  Program  management,  priority-setting,  and 
decision  making.  The  scheduled  completion  date  for  our  work  is  01  September  1993.  After  that 
time,  Mr  Gallogly  and  I  would  be  very  happy  to  said  you  an  executive  summary  of  our  work. 


H-2 


5.  If  you  have  any  questions  or  concerns  about  this  research,  please  do  not  hesitate  to  contact  my 
thesis  advisor.  Dr.  Yupo  Chan,  at  DSN  785-2549  /  Commercial  5 13-255-3362.  Once  again,  thank 
you  very  much  for  your  assistance. 


Thomas  E.  Ellingson 
AFIT  Graduate  Student 


2  Atch 

1.  Questionnaire  #3 

2.  Round  Two  Summary  of  Results 


Attachment  1 


Development  of  Factor  Utility  in  Prioritizing  1RP  Projects 
Questionnaire  #3 


Name: 


The  following  set  of  questions  is  intended  to  determine  your  utility  for  specific  values  within 
each  of  the  criteria  which  were  determined  in  the  previous  two  iterations  of  Delphi  questionnaires. 
The  Summary  of  Results  for  Round  Two  is  included  as  Attachment  2,  and  it  details  how  individual 
respondents  weighted  each  of  the  criteria. 

Please  take  a  few  minutes  to  examine  each  of  the  questions  before  beginning.  Then  return  to 
the  beginning  of  the  questionnaire  and  assign  utility  values  to  the  given  data  points  on  quesuons 
which  require  input.  Please  use  a  scale  of  0  to  100  when  assigning  your  utility  values.  A  utility 
score  of  100  would  indicate  a  situation  within  the  criterion  parameters  which  would  most  strongly 
qualify  a  site  for  immediate  attention;  a  utility  score  of  0  would  indicate  a  situation  within  the 
criterion  parameters  which  would  most  strongly  discourage  a  site  from  receiving  immediate 
attention.  Please  be  sure  to  take  into  account  the  entire  range  of  all  possible  data  points  within  a 
particular  criterion  when  assigning  your  utility  values. 

As  an  example  of  a  possible  response,  the  utility  fix  net  present  worth  of  total  remediation 
effort  cost  might  look  like  the  following: 

Cost: 


Data  Pofet 

Utility 

Total  Site 

$50,000 

-  95 

Remediation 

$200,000 

-  90 

Cost: 

$500,000 

-  85 

$1,000,000 

-  70 

$2,000,000 

-  50 

$4,000,000 

-  35 

$10,000,000 

-  15 

Please  note  that  questions  2  and  5  do  not  require  input,  as  they  are  binary  choice  criteria.  For 
these  criteria,  a  'yes'  response  will  be  scored  as  a  utility  of  100  (best  case),  and  a  'no'  response  will 
be  scored  as  a  0  (worst  case). 

Also,  please  note  that  it  is  not  the  intent  of  this  set  of  questions  to  identify  die  highest  or 
lowest  feasible  values  for  any  particular  criterion  being  considered.  Therefore,  it  is  not  expected 
that  the  first  data  point  should  necessarily  be  given  a  value  of  100,  nor  that  the  last  be  given  a 
value  of  0.  This  set  of  questions  is  simply  attempting  to  extract  an  individual  utility  for  a  set  of 
data  points  which  fell  in  the  typical  range  of  values  for  a  particular  criterion.  Following  this  step, 
the  individual  utility  data  points  will  be  combined  into  a  composite  group  utility  function  which 
will  be  used  as  the  basis  for  the  decision  support  system. 
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1.  Threat  to  Human  Health  and  the  Environment: 

The  relative  risk  presented  by  a  site  to  human  health  or  the  environment  Measured  by 
Defense  Priority  Model  (DPM)  score. 

Data  Point  Utility 

DPM  Score:  50  - _ 

40  -  _ 

35  -  _ 

30  -  _ 

25  -  _ 

20  -  _ 

10  -  • 


2.  Technical  Feasibility  of  Site  Remediation: 

The  availability  of  an  acceptable,  proven  remediation  technology  which  will  meet  the  site- 
specific  applicable  or  relevant  and  appropriate  requirements  (AJRARs)  Measured  in  binary 
(yes/no),  by  availability. 

This  criterion  will  be  measured  by  assigning  a  utility  of  100  if  site  remediation  is  feasible,  or 
a  utility  of  0  otherwise.  Determination  of  availability  is  to  be  made  by  the  Technical  Review 
Committee. 


3.  Cost: 

The  total  estimated  lifetime  cost  of  all  projects  associated  with  the  site.  This  includes  project 
costs  associated  with  implementing  and  maintaining  institutional  controls  (to  meet  ARARs),  such 
as  A/E  design  costs  and  remedial  action  costs;  costs  of  potential  delays  if  work  in  progress  is  not 
completed  in  a  timely  manner;  prevention  of  future  costs  through  immediate  action  or  pollution 
prevention;  etc.  Measured  by  net  present  worth  of  the  total  dollar  cost  estimate  over  the  lifetime  of 
the  remediation  effort. 


Data  Point 

Total  Site 

$50,D00 

Remediation 

$200,000 

Cost: 

$500,000 

$1,000,000 

$2,000,000 

$4,000,000 

$10,000,000 

Utility 
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4.  Expected  Reduction  of  Risk: 

The  estimated  amount  of  risk  (actual  or  potential)  which  can  be  reduced  through  remediation 
efforts  This  will  include  both  short-  and  long-term  expected  effectiveness  of  remedial  actions. 
Measured  by  estimated  percent  reduction  in  DPM  score.  Determination  of  the  expected  DPM 
score  reduction  is  to  be  performed  by  the  Technical  Review  Committee. 

Data  Point  Utility 

%  Reduction  90%  -  _ 

of  DPM  Score:  75%  -  _ 

60%  -  _ 

50%  -  _ 

40%  -  _ 

30%  -  _ 

20%  -  _ 


5.  Federal  Facility  Agreement  ( FFA/IAG ): 

The  presence  of  a  Federal  Facility  Agreement  (FFA)  or  an  Interagency  Agreement  (IAG) 
which  specifies  a  procedural  framework  and  time  schedule  for  the  site.  Measured  in  binary 
(yes/no),  by  presence. 

This  criterion  w*"  be  measured  by  assigning  a  utility  of  100  if  an  FFA/IAG  is  in  place  and 
requires  site  remediation,  otherwise,  a  utility  of  0  will  be  assigned. 


6.  Community  Acceptance: 

Degree  to  which  the  local  community,  including  all  stakeholders,  supports  the  proposed 
remediation  effort.  This  will  be  determined  by  communication  with  the  Technical  Review 
Committee  and  Public  Affairs.  Measured  as  high,  medium,  or  low  level  of  support. 

Data  Point  Utility 

High  Support  — _ 

Medium  Support  -  _ 

Low  Support  -  _ 


7.  Mission  Impact: 

The  net  of  all  anticipated  impact  of  site  remediation  efforts  on  the  ability  of  various  base 
organizations  to  perform  their  designated  missions.  Measured  as  varying  degrees  of  impact 
(positive,  no  impact,  or  negative). 

Data  Point  Utility 


Strong  Positive  Impact  — 
Slight  Positive  Impact  — 
No  Impact  — 
Slight  Negative  Impact  - 
Strong  Negative  Impact  — 
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8.  Socioeconomic/Potilical  Impacts: 

The  net  of  all  anticipated  impacts  of  site  remediation  efforts  on  the  quality  of  life,  local 
economy,  and/or  local  politics  in  the  neighboring  areas.  Measured  as  varying  degrees  of  impact 
(positive,  no  impact,  or  negative).  Determination  of  the  anticipated  impact  level  is  to  be  performed 
by  the  Technical  Review  Committee. 

Data  Point  Utility 

Strong  Positive  Impact  -  _ 

Slight  Positive  Impact  — _ 

No  Impact  -  _ 

Slight  Negative  Impact  -  _ 

Strong  Negative  Impact  -  _ 
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&BPendiX-I , _ Analysis  q£  Variance  Calculations 


This  appendix  contains  the  calculations  performed  in 
association  with  the  analysis  of  variance  for  the  Delphi 
weights  and  utilities. 
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This  appendix  contains  the  calculations  performed  in 
association  with  the  MAUT  procedure  to  arrive  at  the 
weighting  factors  for  the  MAUT  multi-criteria  decision 
model . 


Simplified  additive  weight  function: 

Value  (yj.yj.yJ  =  w.vty,)  +  w,v(y3) 

Indifference  between  projects  implies: 
Value  (»)  =  Value  (,), 


therefore: 


w,v(yA1)  +  WjVly^)  =  w,v(y„)  +  w,v(y„) 


Question  5:  DPM,  =  30  =  0.414(30)°"  =  0.67 

DPM.  =  15  U*.  =  0.414(15)°"  =  0.35 

Cost.  =  $500,000  UCo«  =  $0.5M/$10M  =0.05 

(Utility  for  Cost,  can  be  assumed  to  be  1.0 
sin^e  decision  maker  always  preferred  site  A 
regardless  of  cost . ) 


Question  14.1:  DPM.  =  15  U^  =  0.35 

DPM,  =  35  =  0.76 

Difference  in  utility  for  cost  can  be  assumed 
to  be  1.0  since  the  decision  maker  always 
prefers  site  B  regardless  of  difference  in 
cost . 


w,(0.76)  +  w,(0)  >  w,(0.38)  +  w, (1) 
also,  w,  +  w3  =  1 

Simultaneous  solution  yields:  0  1  w,  <  0.27 

0.73  £  w,  £  1 

Question  14.2:  DPM,  =  25  U„„  =  0.58 

DPM,  =  35  U*.  =  0.76 

w,  ( 0 . 7 6 )  +  Wj(0)  >  w,  (0 . 58)  +  w,(l) 
also,  w,  +  Wj  =  1 

Simultaneous  solution  yields:  0  £  w,  <  0.15 

0.85  £  w,  £  1 
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Given  the  three  solutions,  we  can  say  that  0.85  <  w,  <1 
is  the  only  solution  that  holds  for  all  three  conditions. 

The  decision  maker  was  asked  to  further  interpret  these 
results  to  pin  down  a  final,  single  value.  The  decision 
maker  indicated  that  he  would  set  w,  =1  and  let  w,=0 .  In 
other  words,  he  would  not  let  a  difference  in  cost  influence 
his  decision  when  there  existed  a  difference  in  site  risk. 
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